Transistors. Under the covers of eBay’s big data operation — Cloud Computing News. Payments. API. Analytics. POS. ParkFree. NoSQL Pioneers Are Driving the Web's Manifest Destiny. Twitter has scaled back its plans to store billions of tweets using Cassandra, a non-relational database project that Facebook created and open sourced.
Friday night, Twitter said that it will still use Cassandra in a new real-time analytics project it is building, but the decision to move away from plans to migrate tweets from its current MySQL database to Cassandra is seen by some as a blow to startups and open-source projects that are attempting to move beyond relational databases. But in reality, the level of interest about what database architecture some popular startup is using goes beyond Twitter and Cassandra, and touches on the changing nature of both the web and the software that underlies it. In short, the story here isn’t about Cassandra or databases themselves, but about groups of pioneering programmers reacting to the new ways they can build software in a world where computing is cheap.
Cassandra NoSQL Database an Apache Top Level Project. After Facebook made the Cassandra project open source in 2008, the highly scalable, non-relational distributed database proved its mettle at other companies including Cisco, Digg, and Rackspace.
Now the well-known NoSQL database has proven itself as an active Apache project with enough training to graduate from the incubator. The board recently approved a resolution to adopt Cassandra as a Top Level Project. The new "Apache Cassandra" should serve as another example of a high-profile, non-relational data solution and success story Cassandra was born out of Facebook's need to store reverse indices of Facebook messages that users send and receive while communicating with their friends.
The solution needed to scale incrementally while remaining cost effective. Data Model Every row is identified by a unique key. In the most recent version of Cassandra has improved concurrency across the board (see above). Saying Yes to NoSQL; Going Steady with Cassandra. Welcome to Digg Digg delivers the most interesting and talked about stories on the Internet right now.
The Internet is full of great stories, and Digg helps you find, read, and share the very best ones. Cassandra @ Twitter: An Interview with Ryan King. There have been confirmed rumors about Twitter planning to use Cassandra for a long time.
But except the mentioned post, I couldn’t find any other references. Twitter is fun by itself and we all know that NoSQL projects love Twitter. So, imagine how excited I was when after posting about Cassandra 0.5.0 release, I received a short email from Ryan King, the lead of Cassandra efforts at Twitter simply saying that he would be glad to talk about these efforts. How Twitter Uses NoSQL - ReadWriteCloud. InfoQ has released a video of Twitter's Kevin Weil speaking at Strange Loop earlier this year on how the company uses NoSQL.
Weil is quick to point out that Twitter is heavily dependent on MySQL. However, Twitter does employ NoSQL solutions for many purposes for which MySQL isn't ideal. NSA open sources Google database mimic. High performance access to file storage The US National Security Agency is open sourcing a distributed "NoSQL" database based on Google's proprietary BigTable platform.
Known as Accumulo, the platform has been in development at the NSA for over three years, and it's built atop Hadoop, the open source distributed file system and distributed number-crunching platform that mimics Google's internal infrastructure. Unlike existing BigTable mimics such as HBase, Accumolo has "fine-grained" access controls and a new server-side programming mechanism that can modify data that's written to disk, or returned to the user. Using the cell-level access labels, you can provide external servers with access to some cells in the Accumolo data store but not others. The NSA believes this may be of interest to government and health care operations and other outfits concerned with privacy.
Talk on eBay architecture. Randy Shoup and Dan Pritchett gave a talk on scaling eBay, "The eBay Architecture", at SD Forum 2006.
The slides are available (PDF). The parallels with Amazon are remarkable. Like Amazon, eBay started with a two-tiered architecture. Like Amazon, they split the website into a cluster in the late 1990's, followed soon after by partitioning the databases. Like Amazon, they soon encountered poor performance and difficulty compiling their massive, monolithic binary (150M for eBay, Randy and Dan say). They even built their own search engine because "no off-the-shelf search engine met [their] needs. "
Amazon Architecture. This is a wonderfully informative Amazon update based on Joachim Rohde's discovery of an interview with Amazon's CTO.
You'll learn about how Amazon organizes their teams around services, the CAP theorem of building scalable systems, how they deploy software, and a lot more. Many new additions from the ACM Queue article have also been included. Amazon grew from a tiny online bookstore to one of the largest stores on earth. They did it while pioneering new and interesting ways to rate, review, and recommend products. Amazon Technology" Did Rails Sink Twitter? Twitter is arguably the most heavily used Ruby on Rails application in the world.
Almost since its inception, Twitter has fostered a wildly passionate cult following. Also from the beginning, Twitter has suffered from chronic outages under that load. In the past month, record downtime has prompted fresh outcry within its ever-growing user base. This, along with increased attention from mainstream media has forced Twitter to publicly acknowledge these issues, and to reveal a few details about the source of these problems.