background preloader

Scaling Pinterest - From 0 to 10s of Billions of Page Views a Month in Two Years

Scaling Pinterest - From 0 to 10s of Billions of Page Views a Month in Two Years
Pinterest has been riding an exponential growth curve, doubling every month and half. They’ve gone from 0 to 10s of billions of page views a month in two years, from 2 founders and one engineer to over 40 engineers, from one little MySQL server to 180 Web Engines, 240 API Engines, 88 MySQL DBs (cc2.8xlarge) + 1 slave each, 110 Redis Instances, and 200 Memcache Instances. Stunning growth. So what’s Pinterest's story? To tell their story we have our bards, Pinterest’s Yashwanth Nelapati and Marty Weiner, who tell the dramatic story of Pinterest’s architecture evolution in a talk titled Scaling Pinterest. This is the talk they would have liked to hear a year and half ago when they were scaling fast and there were a lot of options to choose from. This is a great talk. Two of my favorite lessons from the talk: Architecture is doing the right thing when growth can be handled by adding more of the same stuff. These two lessons are interrelated. Now, let’s see how Pinterest scales: Basics Solr

Related:  Server-sidePinterestweb server development

Twitter Architecture Toy solutions solving Twitter’s “problems” are a favorite scalability trope. Everybody has this idea that Twitter is easy. With a little architectural hand waving we have a scalable Twitter, just that simple. Well, it’s not that simple as Raffi Krikorian, VP of Engineering at Twitter, describes in his superb and very detailed presentation on Timelines at Scale. If you want to know how Twitter works - then start here. Is your Business Right for Pinterest? 5 Ways to Mold it to the New Trend Pinterest is an online pin board that allows you to share photos and post them to your own pin board. Other users can share your pins or pin your pins to their boards. The result is having one pin being seen by hundreds or even thousands of people. Pinterest is a fun and visual way to gain new ideas and share your favorites with others. Pinterest can also be an excellent marketing tool and allow you to expand your customer base and gain exposure.

How we built our Real-time Analytics Platform » MaxCDN Blog With the recent release of our Analytics Platform, we would like to give you a behind the scenes look of how we built it. The amount of log data our worldwide network produces is staggering. Every day, our servers generate over 8 terabytes of raw log data (before compression). During our peak times, our servers are generating over 200,000 requests per second. Pinterest Campaigns: Effective Or Overhyped Even though Pinterest is not compatible with every brand because of its visual nature, narrow demographics (users are mainly women between the ages of 18 and 35), and a lack of opportunities to start a conversation with followers, several notable brand campaigns have arisen. One of the most important strengths of Pinterest is the possibility to hit on people’s emotions through the images that are published, according to Erin Houg, VP and digital director at Starcom. And that is exactly what organizations and brands like UNICEF and South Africa’s Nando’s are trying to do. But are they effectively doing so, or are they hopping on the Pinterest bandwagon too soon? UNICEF recently launched a campaign on Pinterest to raise awareness for children in need, by posing as 13-year-old Ami Musa from Sierra Leone. Each pin contains a link leading to a donation page set up by UNICEF, and has been repinned at least 100 times.

Large Scale Web Site Development TSS recently had an active thread on Why most large-scale Web sites are not written in Java. This is a provocative title and naturally caused a lot of passionate readers comment on this thread. The thread was started in response to a posting by Nati Shalom, which seems to have been prompted by a similar posting at While I don’t disagree with the analysis presented in Nati Shalom’s post, I don’t think Java, the programming language, or JEE is to be blamed. The key driver behind most successful large scale web sites is that they are designed to perform, often taking the most unorthodox approaches possible towards scalability, and most JEE developers have neither the incentive nor the experience to make their sites scalable. The techniques used by large scale web sites often seem bizarre to JEE web developers.

Solving Problems With Spring Boot I first became aware of Spring Boot early in 2013. At the time, we were rebuilding Spring's website and decided to write a collection of guides that could be consumed during a single lunch break (at Realizing how much code Spring Boot saved us from writing (and explaining to readers) thanks to its auto-configuration feature, we embraced it fully. In fact, it led me to write several patches for Spring Boot to help with several of the guides I was writing. Discovering that boiler plate Spring code was unnecessary was incredibly exciting and very effective. Why Pinterest Overtakes Facebook In Social Commerce Research shows that online retailers prefer to use the Pinterest ‘Pin It’ button on their goods rather than the old faithful Facebook ‘Like’. The third annual Social Commerce IQ (SCIQ) report made by 8thBridge analyzes the social commerce strategies of 872 retailers. The number of retailers analyzed this year was almost double the number analyzed in last year’s report, to include nearly all of the brands from companies in the Internet Retailer 2013 Top 500 Guide and additional brands Internet Retailer determined were up and coming. Wade Gerten, CEO of 8thBridge, Inc. stated: 7 social platforms were analyzed for the 872 retailers including: Facebook, YouTube, Twitter, Instagram, Pinterest, Google+, and Vine. 4 areas were taken into account when calculating the SCIQ scores: brand awareness on social networks, social upstream traffic, website social lift, and Social CRM.

Why most large-scale Web sites are not written in Java I think that looks pretty nice. But it looks nothing like what I have to work with. I'm in 'how do I get from there to here' hell. I'm in the actions, then the jsps, then struts config and back to the jsp to look at some javascript and then to some beans etc. By the time I find what I'm looking for, I can't remember why I was looking for it. Node.js in Production When running a node application in production, you need to keep stability, performance, security, and maintainability in mind. Outlined here is what I think are the best practices for putting node.js into production. By the end of this guide, this setup will include 3 servers: a load balancer (lb) and 2 app servers (app1 and app2).

Skimlinks Releases Full API For Web Publishers Skimlinks, a content monetization platform for web publishers, has released a comprehensive API suite. Part of this allows publishers to learn more about their users’ shopping habits. The startup made the news recently when it was briefly used by hot startup Pinterest to monetize its outgoing links – although that arrangement foundered after it attracted controversy. The four APIs include: Link, Merchant, Reporting, and Product. Link API enables automatic link “affiliatization” using a developer’s custom redirect scripts, or on apps where they cannot run the standard Skimlinks javascript. Merchant API lets publishers search and filter the Skimlinks’ merchant list by category and name.

Node.js w/1M concurrent connections! I’ve decided to ramp up the Node.js experiments, and pass the 1 million concurrent connections milestone. It worked, using a swarm of 500 Amazon EC2 test clients, each establishing ~2000 active long-poll COMET connections to a single 15GB rackspace cloud server. This isn’t landing the mars rover, or curing cancer. It’s just a pretty cool milestone IMO, which may help a few people who want to use Node.js for a large number of concurrent connections. So, hopefully it’s of some benefit to a few Node developers who can use these settings as a starting point in their own projects. Here’s the connection count as displayed on the sprite’s page:

Is Pinterest the next Facebook? An exclusive look inside CEO Ben Silbermann's social media sensation. FORTUNE -- Ben Silbermann can't stop staring at the refrigerators. The Pinterest co-founder and CEO and I are standing in the break room of his company's garage-size Palo Alto office. He's just flown back from Austin's SXSW interactive festival, and a redesign of his website is two days away. The Instagram Architecture Facebook Bought for a Cool Billion Dollars It's been a well kept secret, but you may have heard Facebook will Buy Photo-Sharing Service Instagram for $1 Billion. Just what is Facebook buying? Here's a quick gloss I did a little over a year ago on a presentation Instagram gave on their architecture. In that article I called Instagram's architecture the "canonical description of an early stage startup in this era." Little did we know how true that would turn out to be. If you want to learn how they did it then don't take a picture, just keep on reading...

Hardening node.js for production: a process supervisor This post outlines a basic process supervisor for node.js that can be used to manage multiple PIDs underneath the same application. It contains sample code for our child_monitor with a deep health check for socket servers. We started using node.js in the Silly Face Society because it seemed like the quickest way to get a (true, not web) socket server up for our “party mode”. The party mode is stateful chat+game server where a set of processes check into redis to advertise themselves as available to host games (in another post I’ll expand on this architecture). For development, node.js was a wonderful way to get a prototype out the door but as our launch approaches are getting nervous the robustness of node.js in production.