Evolution of SoundCloud's Architecture SoundCloud Backstage. August 30th, 2012 This is a story of how we adapted our architecture over time to accomodate growth.
Scaling is a luxury problem and surprisingly has more to do with organization than implementation. For each change we addressed the next order of magnitude of users we needed to support, starting in the thousands and now we’re designing for the hundreds of millions. We identify our bottlenecks and addressed them as simply as possible by introducing clear integration points in our infrastructure to divide and conquer each problem individually. By identifying and extracting points of scale into smaller problems and having well defined integration points when the time arrived, we are able to grow organically.
From day one, we had the simple need of getting each idea out of our heads and in front of eyeballs as quickly as possible. We consciously chose not to implement high availability at this point, knowing what it would take when that time hopefully arrived. Scaling Pinterest - From 0 to 10s of Billions of Page Views a Month in Two Years. Pinterest has been riding an exponential growth curve, doubling every month and half.
They’ve gone from 0 to 10s of billions of page views a month in two years, from 2 founders and one engineer to over 40 engineers, from one little MySQL server to 180 Web Engines, 240 API Engines, 88 MySQL DBs (cc2.8xlarge) + 1 slave each, 110 Redis Instances, and 200 Memcache Instances. Stunning growth. Large Scale Web Site Development. TSS recently had an active thread on Why most large-scale Web sites are not written in Java.
This is a provocative title and naturally caused a lot of passionate readers comment on this thread. The thread was started in response to a posting by Nati Shalom, which seems to have been prompted by a similar posting at highscalability.com. While I don’t disagree with the analysis presented in Nati Shalom’s post, I don’t think Java, the programming language, or JEE is to be blamed. The key driver behind most successful large scale web sites is that they are designed to perform, often taking the most unorthodox approaches possible towards scalability, and most JEE developers have neither the incentive nor the experience to make their sites scalable.
Why most large-scale Web sites are not written in Java. I think that looks pretty nice.
But it looks nothing like what I have to work with. I'm in 'how do I get from there to here' hell. I'm in the actions, then the jsps, then struts config and back to the jsp to look at some javascript and then to some beans etc. By the time I find what I'm looking for, I can't remember why I was looking for it. I feel that pain. One thing to note too is that there is not a lick of Java in that code. You're absolutely correct on both counts. And yes that does give me some hope.
Our ca. 2000 Struts project no doubt looks exactly like yours. This can become, simply using JSTL: A simple step eliminating little Java bits. The tag prefixes the path for you, and the "border='0'" comes "for free" now. 7 Years of YouTube Scalability Lessons in 30 Minutes. If you started out building a dating site and instead ended up building a video sharing site (YouTube) that handles 4 billion views a day, then it’s just possible you learned something along the way.
And indeed, Mike Solomon, one of the original engineers at YouTube, did learn a lot and he has given a talk about it at PyCon: Scalability at YouTube. This isn’t an architecture driven talk where we are led through a description of how a lot of boxes connect to each other. Mike could give that sort of talk. He has worked on building YouTube’s servlet infrastructure, video indexing feature, video transcoding system, their full text search, a CDN, and much more.
But instead, he’s taken a step back, took a long look around at what time has wrought, and shared some deep lessons, obviously hard won from experience. The key takeaway away of the talk for me was doing a lot with really simple tools. Stats. Mike Krieger, Instagram at the Airbnb tech talk, on Scaling Instagram. Best Practices for Large-Scale Web Sites. The Instagram Architecture Facebook Bought for a Cool Billion Dollars. It's been a well kept secret, but you may have heard Facebook will Buy Photo-Sharing Service Instagram for $1 Billion.
Just what is Facebook buying? Here's a quick gloss I did a little over a year ago on a presentation Instagram gave on their architecture. In that article I called Instagram's architecture the "canonical description of an early stage startup in this era. " Little did we know how true that would turn out to be. If you want to learn how they did it then don't take a picture, just keep on reading... Instagram is a free photo sharing and social networking service for your iPhone that has been an instant success. The Instagram team has written up what can be considered the canonical description of an early stage startup in this era: What Powers Instagram: Hundreds of Instances, Dozens of Technologies. Instagram uses a pastiche of different technologies and strategies. We'll just tl;dr the article here, it's very well written and to the point.