background preloader

Scalability

Facebook Twitter

Facebook trapped in MySQL ‘fate worse than death’ — Cloud Computing News. MongoDB is Web Scale. Why are Facebook, Digg, an. Real-time social graphs (connectivity between people, places, and things).

Why are Facebook, Digg, an

That's why scaling Facebook is hard says Jeff Rothschild, Vice President of Technology at Facebook. Social networking sites like Facebook, Digg, and Twitter are simply harder than traditional websites to scale. Why is that? Why would social networking sites be any more difficult to scale than traditional web sites? Let's find out. Traditional websites are easier to scale than social networking sites for two reasons: Bytepawn - Scalable Web Architectures and Application State. In this article we follow a hypothetical programmer, Damian, on his quest to make his web application scalable.

Bytepawn - Scalable Web Architectures and Application State

Now fast forward to 2009. Damian's site has evolved to a web game for playing dungeons online, in your browser. Damian is still using LAMP. Data about types of games (including parameters such as monster strength), user data (including status information), and data about active games (including players, the monters's health) are still stored in Mysql. It's all fine until there are only a few games in session and only a couple hundred players, but as the site gets popular, Damian's server is starting to see high load numbers. Damian is experiencing scalability issues --- his current setup cannot handle tens of thousands of users. Against all the odds. First let's describe what means by odds: In my social network, I found 93% of the mainstream developers sanctify the database, or at least consider it in any data persistence challenge as the ultimate, superhero, and undefeatable solution. I think this problem come from the education, personally, and some companies also I think it's involved in this.

Are Cloud Based Memory Architectures the Next Big Thing? » Scalable Web Applications Programming the new world: Programmi. Purpose of the entry On Saturday June 13th 2009 I attended a talk by Eli White on Scalable web applications.

» Scalable Web Applications Programming the new world: Programmi

Eli White previously worked at digg.com and now holds the position PHP Community Manager & DevZone Editor-in-Chief at Zend Technologies. Node and Scaling in the Small vs Scaling in the Large. Over the past few weeks, I’ve been taking whatever spare moments I can find to think about what technologies we’re going to use to build the initial release of BankSimple.

Node and Scaling in the Small vs Scaling in the Large

Many people would probably assume that I’d immediately reach for Scala, what with having co-authored a book on the language, but that’s not how I approach engineering problems. Each and every problem has an appropriate set of applicable technologies, and it’s up to the engineer to justify their use. (Incidentally, Scala may well be a good fit for BankSimple, in no small part due to a bunch of third-party Java code that we need to integrate with, but that’s a whole different blog post, probably for a whole different blog.)

One of the most talked-about technologies amongst the Hacker News crowd is Node, a framework for writing and running event-driven JavaScript code on the V8 virtual machine. How to Succeed at Capacity Planning Without Really Trying : An I. Update 2: Velocity 09: John Allspaw, 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr.

How to Succeed at Capacity Planning Without Really Trying : An I

Insightful talk. Some highlights: Change is good if you can build tools and culture to lower the risk of change. Operations and developers need to become of one mind and respect each other. An automated infrastructure is the one tool you need most. Common source control.

Examples

Digg: 4000% Performance In. An Unorthodox Approach to Database Design : The Coming of the Sh. Update 4: Why you don’t want to shard. by Morgon on the MySQL Performance Blog.

An Unorthodox Approach to Database Design : The Coming of the Sh

Optimize everything else first, and then if performance still isn’t good enough, it’s time to take a very bitter medicine. Update 3: Building Scalable Databases: Pros and Cons of Various Database Sharding Schemes by Dare Obasanjo. Excellent discussion of why and when you would choose a sharding architecture, how to shard, and problems with sharding.Update 2: Mr. Moore gets to punt on sharding by Alan Rimm-Kaufman of 37signals. Insightful article on design tradeoffs and the evils of premature optimization. Once upon a time we scaled databases by buying ever bigger, faster, and more expensive machines. What is sharding and how has it come to be the answer to large website scaling problems? Information Sources. 6 Ways to Kill Your Servers - Learning How to Scale the Hard Way. This is a guest post by Steffen Konerow, author of the High Performance Blog.

6 Ways to Kill Your Servers - Learning How to Scale the Hard Way

Learning how to scale isn’t easy without any prior experience. No to SQL? Anti-database movement gains stea. Eric Lai published a provoking article on Computerworld magazine titled “No to SQL?

No to SQL? Anti-database movement gains stea

Anti-database movement gains steam” where he pointed to many references in which different Internet-based companies chose an alternative approach to the traditional SQL database. The write-up was driven from the the inaugural get-together of the burgeoning NoSQL community who seem to represent a growing Anti-SQL database movement. Quoting Jon Travis from this article: Relational databases give you too much. They force you to twist your object data to fit a RDBMS [relational database management system], The article points to specific examples that led different companies such as Google, Amazon, Facebook to choose an alternative approach.

Demand for extremely large scale: “BigTable, is used by local search engine Zvents Inc. to write 1 billion cells of data per day.” High Performance Scalable Data Stores. Advice from Google on large distributed syste. Google Fellow Jeff Dean gave a keynote talk at LADIS 2009 on "Designs, Lessons and Advice from Building Large Distributed Systems".

Advice from Google on large distributed syste

Scaling Online Social Networks without Pains. Real World Web: Performance & Scalability. Put that database in memory. An upcoming paper, "The Case for RAMClouds: Scalable High-Performance Storage Entirely in DRAM" (PDF), makes some interesting new arguments for shifting most databases to serving entirely out of memory rather than off disk.

Put that database in memory

The paper looks at Facebook as an example and points out that, due to aggressive use of memcached and caches in mysql, the memory they use already is about "75% of the total size of the data (excluding images). " 10 eBay Secrets for Planet. You don't even have to make a bid, Randy Shoup, an eBay Distinguished Architect, gives this presentation on how eBay scales, for free. Randy has done a fabulous job in this presentation and in other talks listed at the end of this post getting at the heart of the principles behind scalability. It's more about ideas of how things work and fit together than a focusing on a particular technology stack. Impressive Stats In case you weren't sure, eBay is big, with lots of: users, data, features, and change... Over 89 million active users worldwide190 million items for sale in 50,000 categoriesOver 8 billion URL requests per dayHundreds of new features per quarterRoughly 10% of items are listed or ended every dayIn 39 countries and 10 languages24x7x36570 billion read / write operations / dayProcesses 50TB of new, incremental data per dayAnalyzes 50PB of data per day.

Cassandra @ Twitter: An Interview with Ryan King « MyNoSQL. There have been confirmed rumors about Twitter planning to use Cassandra for a long time. But except the mentioned post, I couldn’t find any other references. Twitter is fun by itself and we all know that NoSQL projects love Twitter. So, imagine how excited I was when after posting about Cassandra 0.5.0 release, I received a short email from Ryan King, the lead of Cassandra efforts at Twitter simply saying that he would be glad to talk about these efforts.

Howfuckedismydatabase.com. Troubles with Sharding - What can we learn from the Foursquare Incident? For everything given something seems to be taken.