background preloader

Scaling Pinterest - From 0 to 10s of Billions of Page Views a Month in Two Years

Scaling Pinterest - From 0 to 10s of Billions of Page Views a Month in Two Years
Pinterest has been riding an exponential growth curve, doubling every month and half. They’ve gone from 0 to 10s of billions of page views a month in two years, from 2 founders and one engineer to over 40 engineers, from one little MySQL server to 180 Web Engines, 240 API Engines, 88 MySQL DBs (cc2.8xlarge) + 1 slave each, 110 Redis Instances, and 200 Memcache Instances. Stunning growth. So what’s Pinterest's story? To tell their story we have our bards, Pinterest’s Yashwanth Nelapati and Marty Weiner, who tell the dramatic story of Pinterest’s architecture evolution in a talk titled Scaling Pinterest. This is a great talk. Two of my favorite lessons from the talk: Architecture is doing the right thing when growth can be handled by adding more of the same stuff. These two lessons are interrelated. Now, let’s see how Pinterest scales: Basics Launched in March 2010 - The Age of Finding Yourself At this point you don’t even know what product you are going to build. January 2011 Why MySQL? Solr Related:  Server-sidePinterestweb server development

Twitter Architecture Toy solutions solving Twitter’s “problems” are a favorite scalability trope. Everybody has this idea that Twitter is easy. With a little architectural hand waving we have a scalable Twitter, just that simple. Well, it’s not that simple as Raffi Krikorian, VP of Engineering at Twitter, describes in his superb and very detailed presentation on Timelines at Scale. It happened gradually so you may have missed it, but Twitter has grown up. Twitter now has 150M world wide active users, handles 300K QPS to generate timelines, and a firehose that churns out 22 MB/sec. 400 million tweets a day flow through the system and it can take up to 5 minutes for a tweet to flow from Lady Gaga’s fingers to her 31 million followers. A couple of points stood out: Twitter no longer wants to be a web app. How does Twitter work? The Challenge Groups The Platform Services group is responsible for the core scalable infrastructure of Twitter. Push Me Pull Me People are creating content on Twitter all the time.

5 Insights on Facebook vs Pinterest in Driving Sales - Infographic Pinterest has been grabbing the headlines recently as its network approaches 20 million users per month. In March, Compete.com reported that Pinterest had received 18 million unique visitors with AppData reporting that the visual social network had received a record 4.4 million daily active users on March 18. There are reports that Pinterest’s growth is slowing but just like Twitter and Facebook, reports of its demise maybe slightly premature. The Power of “Low Friction” Photo Sharing Pinterest’s value to marketers is that it provides an easy to use platform for “pinning” images, photos and videos to your Pinterest account while also posting and sharing to your networks to Facebook and Twitter in the same process. Pinterest reduces the effort and friction to share images on the social web which crowd sources your marketing and the spreading of your brand images. This “low friction sharing” includes links back to the original sharing location. The question though is, does it drive sales?

Runtime.exec() pour les nuls et ProcessBuilder Il vous est déjà sûrement arrivé, un jour, de vouloir exécuter un application externe à partir de votre programme Java. Et si c’est le cas, vous avez très probablement eu du mal à faire fonctionner votre programme correctement, celui-ci semblant se bloquer et ne plus rien afficher. Dans ce petit article, je vais vous expliquer rapidement comment éviter ce genre de problème avec le classic Runtime.exec(), et je vous décrirai ensuite son substitut, le ProcessBuilder. Runtime.exec() Erreur n°1 : où est le waitFor() ? Supposons que vous ayez un petit script batch (ou shell) qui va afficher « Hello World! packagecom.excilys.labs; import java.io.IOException; public class Main { public static final String CHEMIN = "C:\\workspace\\"; Et là, sur votre écran, vous obtenez un magnifique : Début du programme Fin du programme Vous ne comprenez pas pourquoi rien ne s’affiche, mais vous ne paniquez pas. Vous modifiez donc un peu votre code : A l’exécution, vous obtenez simplement : Vous commencez à paniquer…

How we built our Real-time Analytics Platform » MaxCDN Blog With the recent release of our Analytics Platform, we would like to give you a behind the scenes look of how we built it. The amount of log data our worldwide network produces is staggering. Every day, our servers generate over 8 terabytes of raw log data (before compression). Handle Current And Future Insert VolumeHandle Unstructured Request DataMinimal Hardware with Redundancy And ScaleMinimal Software LayersImmediate Query Capabilities For Customers We wanted to rebuild our reporting and analytics platform to provide more informative metrics to our customers to assist them in making smart business decisions when it comes to their online content. Because of our unique requirements, we could not simply pick something up off the shelf and plug it in. In order to insert data as quickly as possible into TokuMX, we wrote custom agents in Go. The bridge between our CDN and our database cluster is powered by a high-volume messaging queue built using Redis. Want to engineer things like this?

How To Use Pinterest’s Group Boards To Get More Exposure For Your Business In case you hadn’t noticed, Pinterest has been in the news a lot recently. In November last year, Pinterest started courting the business community with the creation of its business-specific accounts and a dedicated business support page. Then earlier this month, Pinterest quietly raised a cool $200 million at an astonishing $2.5 billion valuation – not bad for a company that has yet to make a profit! Most intriguing of all though was a study published last week by research center Pew. It suggested that Pinterest has grown so quickly, it’s now on course to catch Twitter in the battle for second place behind Facebook in the US social networking market. These events indicate that Pinterest is quickly evolving from being last year’s hot new upstart to a legitimate social network that should be part of your marketing mix. How can you maximize your time on Pinterest? But how can you maximize your Pinterest activities without spending a ton of time on it? What Are Pinterest’s Group Boards? #1. #2.

Large Scale Web Site Development TSS recently had an active thread on Why most large-scale Web sites are not written in Java. This is a provocative title and naturally caused a lot of passionate readers comment on this thread. The thread was started in response to a posting by Nati Shalom, which seems to have been prompted by a similar posting at highscalability.com. While I don’t disagree with the analysis presented in Nati Shalom’s post, I don’t think Java, the programming language, or JEE is to be blamed. The key driver behind most successful large scale web sites is that they are designed to perform, often taking the most unorthodox approaches possible towards scalability, and most JEE developers have neither the incentive nor the experience to make their sites scalable. For the record, I have worked for over seven years for BEA Systems until I joined Yahoo about a month ago, and have great respect and admiration for BEA’s middleware stack. Sessions: HTTP sessions don’t, in general, scale. One last point.

Solving Problems With Spring Boot I first became aware of Spring Boot early in 2013. At the time, we were rebuilding Spring's website and decided to write a collection of guides that could be consumed during a single lunch break (at spring.io/guides). Realizing how much code Spring Boot saved us from writing (and explaining to readers) thanks to its auto-configuration feature, we embraced it fully. In fact, it led me to write several patches for Spring Boot to help with several of the guides I was writing. Another of Spring Boot's amazing features was its support for properties. Another golden feature is Spring Boot's library of built-in actuators. Every feature Spring Boot provides is incredibly appealing to quickly moving into coding features and deploying them into real apps. I was speaking with a colleague a couple months ago and discovered he was helping a local school to setup a computer science workshop. So taking all this into account, I started writing my proposal earlier this year about Spring Boot.

How Can Your Business Benefit From Pinterest’s New Analytics Tool? Last week Pinterest finally launched what many business users on the image-sharing network have been crying out for – its brand new web analytics. Why all the excitement? Well up until now, Pinterest users have found it difficult to gain meaningful insights into how their audience was engaging with their content. Whilst you could check new followers, likes, repins and comments via the activity feed, unless you were using third-party solutions like the excellent Curalate, much of the tracking available within Pinterest involved guesswork and assumptions. Although they’re nowhere near the sophistication of Facebook Insights, Pinterest’s analytics are definitely a step in the right direction for bloggers, businesses and organizations wanting to find out what people are pinning from their websites. Let’s take a look at how you can make this new tool work for you. Get Your Page Ready For Pinterest Analytics So let’s get started. #1 Verify Your Page #2 Activate Pinterest’s New Look Key Takeaways

Why most large-scale Web sites are not written in Java I think that looks pretty nice. But it looks nothing like what I have to work with. I'm in 'how do I get from there to here' hell. I'm in the actions, then the jsps, then struts config and back to the jsp to look at some javascript and then to some beans etc. I feel that pain. One thing to note too is that there is not a lick of Java in that code. You're absolutely correct on both counts. And yes that does give me some hope. Our ca. 2000 Struts project no doubt looks exactly like yours. This can become, simply using JSTL: A simple step eliminating little Java bits. The tag prefixes the path for you, and the "border='0'" comes "for free" now.

Node.js in Production When running a node application in production, you need to keep stability, performance, security, and maintainability in mind. Outlined here is what I think are the best practices for putting node.js into production. By the end of this guide, this setup will include 3 servers: a load balancer (lb) and 2 app servers (app1 and app2). The load balancer will health check and balance traffic between the servers. The app servers will be using a combination of systemd and node cluster to load balance and route traffic around multiple node processes on the server. Deploys will be a one-line command from the developer’s laptop and cause zero downtime or request failures. It will look roughly like this: Photo credit: Digital Ocean How this article is written This article is targeted to those with beginning operations experience. The final app is hosted here: For this guide I will be using Digital Ocean and Fedora. Why Fedora? Install Node.js Create web user

Pinterest takes a “first step” toward working with big brands Pinterest plans to announce a new type of pin on Sunday that will highlight a large number of major U.S. retail brands, marking the company’s “first step” in integrating images with associated brands, and making it easier to click through links and purchase items. The move could be the start of a change in consumer perception of the site from a place for wishful thinking to a site where one can purchase those wishes. While this is just a first step, and the company said it is not currently making money from the integration, the power of the companies joining at launch suggests Pinterest could become on par with Facebook and Twitter when it comes to attracting sizable brand marketing budgets in the not-so-distant future. Pinterest is launching three distinct types of pins, with one type each for food, retail products, and movies. The new format will only work with items pinned from the launch partner sites, but the number of partners is wide-reaching, and will grow.

How can I use Python for large scale development Node.js w/1M concurrent connections! | caustik's blog I’ve decided to ramp up the Node.js experiments, and pass the 1 million concurrent connections milestone. It worked, using a swarm of 500 Amazon EC2 test clients, each establishing ~2000 active long-poll COMET connections to a single 15GB rackspace cloud server. This isn’t landing the mars rover, or curing cancer. It’s just a pretty cool milestone IMO, which may help a few people who want to use Node.js for a large number of concurrent connections. So, hopefully it’s of some benefit to a few Node developers who can use these settings as a starting point in their own projects. Here’s the connection count as displayed on the sprite’s page: Here’s a sysctl dumping the number of open file handles (sockets are file handles): Here’s the view of “top” showing system resources in use: I think it’s pretty reasonable for 1M connections to consume 16GB of memory, but it could probably be trimmed down quite a bit. Here’s a latency test run against the comet URL: Leave a Reply

The Pinterest Roadmap Revealed The rise of Pinterest wasn’t all cocktails and cupcakes. It conquered huge scaling challenges to become the world’s keepsake box. Now as the we ditch the desktop for mobile, how will Pinterest evolve to connect us with the things we love? Today Pinterest’s head of engineering Jon Jenkins gave a surprisingly candid look at the past and future of Pinterest journey to build the interest graph. Now Pinterest is a team of 140, with 70 engineers, $338 million in funding, and a massive headquarters in San Francisco. 2010: “The Year Of The Creation Of The Business” Pinterest began as a few guys with a mission “To help people discover the things they love and then do those things in real life. 2011: “The Year Of Scaling Challenges” Jenkins says that “traffic was doubling every month and half” and provided this traffic graph, though with no Y axis about exactly how many page views the site receives. 2012: “The Big Year Of Mobile” 2013: “The Year Of Adding A Lot More Value To Pins” The Future

Related: