background preloader

Scaling Pinterest - From 0 to 10s of Billions of Page Views a Month in Two Years

Scaling Pinterest - From 0 to 10s of Billions of Page Views a Month in Two Years
Pinterest has been riding an exponential growth curve, doubling every month and half. They’ve gone from 0 to 10s of billions of page views a month in two years, from 2 founders and one engineer to over 40 engineers, from one little MySQL server to 180 Web Engines, 240 API Engines, 88 MySQL DBs (cc2.8xlarge) + 1 slave each, 110 Redis Instances, and 200 Memcache Instances. Stunning growth. So what’s Pinterest's story? To tell their story we have our bards, Pinterest’s Yashwanth Nelapati and Marty Weiner, who tell the dramatic story of Pinterest’s architecture evolution in a talk titled Scaling Pinterest. This is a great talk. Two of my favorite lessons from the talk: Architecture is doing the right thing when growth can be handled by adding more of the same stuff. These two lessons are interrelated. Now, let’s see how Pinterest scales: Basics Launched in March 2010 - The Age of Finding Yourself At this point you don’t even know what product you are going to build. January 2011 Why MySQL? Solr Related:  Server-sidePinterestweb server development

Twitter Architecture Toy solutions solving Twitter’s “problems” are a favorite scalability trope. Everybody has this idea that Twitter is easy. With a little architectural hand waving we have a scalable Twitter, just that simple. Well, it’s not that simple as Raffi Krikorian, VP of Engineering at Twitter, describes in his superb and very detailed presentation on Timelines at Scale. It happened gradually so you may have missed it, but Twitter has grown up. Twitter now has 150M world wide active users, handles 300K QPS to generate timelines, and a firehose that churns out 22 MB/sec. 400 million tweets a day flow through the system and it can take up to 5 minutes for a tweet to flow from Lady Gaga’s fingers to her 31 million followers. A couple of points stood out: Twitter no longer wants to be a web app. How does Twitter work? The Challenge Groups The Platform Services group is responsible for the core scalable infrastructure of Twitter. Push Me Pull Me People are creating content on Twitter all the time.

Google+ : bienvenue dans la Matrice La lecture de la semaine est un post du blog techno publié sur le site du Guardian, par Charles Arthur (@charlesarthur), journaliste. Son titre : "Google+ n'est pas un réseau social, c'est Matrix". "Presque tout le monde (moi inclus, dit Charles Arthur) a mal compris ce qu'était Google+. Réfléchissez à ceci : si vous créez un compte Gmail, vous aurez automatiquement un compte Google+. La raison pour laquelle Google+ n'est pas un réseau social, c'est que le fait d'avoir des amis ou de suivre des gens n'est qu'une excroissance accidentelle de son but véritable : être une paroi invisible entre vous et le web, une paroi qui voit ce que vous faites, l'enregistre et le stocke pour un usage futur. C'est là que la Matrice entre en jeu. Bien sûr, poursuit Charles Arthur, dans le monde post-Google+, les résultats les plus pertinents sont de plus en plus ceux qui pointent vers des contenus qui sont la propriété de Google. Pour mieux comprendre, Charles Arthur invite à passer par les métaphores.

5 Insights on Facebook vs Pinterest in Driving Sales - Infographic Pinterest has been grabbing the headlines recently as its network approaches 20 million users per month. In March, Compete.com reported that Pinterest had received 18 million unique visitors with AppData reporting that the visual social network had received a record 4.4 million daily active users on March 18. There are reports that Pinterest’s growth is slowing but just like Twitter and Facebook, reports of its demise maybe slightly premature. The Power of “Low Friction” Photo Sharing Pinterest’s value to marketers is that it provides an easy to use platform for “pinning” images, photos and videos to your Pinterest account while also posting and sharing to your networks to Facebook and Twitter in the same process. Pinterest reduces the effort and friction to share images on the social web which crowd sources your marketing and the spreading of your brand images. This “low friction sharing” includes links back to the original sharing location. The question though is, does it drive sales?

Runtime.exec() pour les nuls et ProcessBuilder Il vous est déjà sûrement arrivé, un jour, de vouloir exécuter un application externe à partir de votre programme Java. Et si c’est le cas, vous avez très probablement eu du mal à faire fonctionner votre programme correctement, celui-ci semblant se bloquer et ne plus rien afficher. Dans ce petit article, je vais vous expliquer rapidement comment éviter ce genre de problème avec le classic Runtime.exec(), et je vous décrirai ensuite son substitut, le ProcessBuilder. Runtime.exec() Erreur n°1 : où est le waitFor() ? Supposons que vous ayez un petit script batch (ou shell) qui va afficher « Hello World! packagecom.excilys.labs; import java.io.IOException; public class Main { public static final String CHEMIN = "C:\\workspace\\"; Et là, sur votre écran, vous obtenez un magnifique : Début du programme Fin du programme Vous ne comprenez pas pourquoi rien ne s’affiche, mais vous ne paniquez pas. Vous modifiez donc un peu votre code : A l’exécution, vous obtenez simplement : Vous commencez à paniquer…

Evolution of SoundCloud's Architecture SoundCloud Backstage August 30th, 2012 This is a story of how we adapted our architecture over time to accomodate growth. Scaling is a luxury problem and surprisingly has more to do with organization than implementation. By identifying and extracting points of scale into smaller problems and having well defined integration points when the time arrived, we are able to grow organically. From day one, we had the simple need of getting each idea out of our heads and in front of eyeballs as quickly as possible. Apache was serving our image/style/behavior resources, and Rails backed by MySQL provided an environment where almost all of our product could be modeled, routed and rendered quickly. We consciously chose not to implement high availability at this point, knowing what it would take when that time hopefully arrived. Our primary cost optimization was for opportunity, and anything that got in the way of us developing the concepts behind SoundCloud were avoided. I heartily recommend Neil J.

How we built our Real-time Analytics Platform » MaxCDN Blog With the recent release of our Analytics Platform, we would like to give you a behind the scenes look of how we built it. The amount of log data our worldwide network produces is staggering. Every day, our servers generate over 8 terabytes of raw log data (before compression). Handle Current And Future Insert VolumeHandle Unstructured Request DataMinimal Hardware with Redundancy And ScaleMinimal Software LayersImmediate Query Capabilities For Customers We wanted to rebuild our reporting and analytics platform to provide more informative metrics to our customers to assist them in making smart business decisions when it comes to their online content. Because of our unique requirements, we could not simply pick something up off the shelf and plug it in. In order to insert data as quickly as possible into TokuMX, we wrote custom agents in Go. The bridge between our CDN and our database cluster is powered by a high-volume messaging queue built using Redis. Want to engineer things like this?

Trois raisons pour Facebook d’aimer les #hashtags La rumeur courait depuis quelques mois, l'annonce officielle est tombée mercredi 12 juin : Facebook va commencer à déployer sur son site américain les "hashtags" (en français officiel : "mot-dièse") dans le statut de ses utilisateurs. Pour beaucoup d'inscrits sur ce réseau social, la nouvelle a d'abord eu de quoi intriguer. "C'est quoi les hashtags ?", nous ont demandé nos lecteurs sur la page Facebook du Monde.fr, question à laquelle d'autres membres ont parfaitement répondu : D'autres de nos lecteurs sur Facebook, fins connaisseurs du "dièse" (le fameux "#"), ne semblent toutefois pas vraiment convaincus de son utilité (voir : #leshashtagscestridicule, #leshashtagscestbidon). 1. Facebook l'avoue sans complexe dans son communiqué de presse : son système sera "similaire" à celui qu'on trouve sur "Instagram, Twitter, Tumblr, ou Pinterest" (sans compter Vine et Google+, également remplis de "#"). En six ans, les mots-clés sont devenus emblématiques des réseaux sociaux. 2. 3.

How To Use Pinterest’s Group Boards To Get More Exposure For Your Business In case you hadn’t noticed, Pinterest has been in the news a lot recently. In November last year, Pinterest started courting the business community with the creation of its business-specific accounts and a dedicated business support page. Then earlier this month, Pinterest quietly raised a cool $200 million at an astonishing $2.5 billion valuation – not bad for a company that has yet to make a profit! Most intriguing of all though was a study published last week by research center Pew. It suggested that Pinterest has grown so quickly, it’s now on course to catch Twitter in the battle for second place behind Facebook in the US social networking market. These events indicate that Pinterest is quickly evolving from being last year’s hot new upstart to a legitimate social network that should be part of your marketing mix. How can you maximize your time on Pinterest? But how can you maximize your Pinterest activities without spending a ton of time on it? What Are Pinterest’s Group Boards? #1. #2.

Large Scale Web Site Development TSS recently had an active thread on Why most large-scale Web sites are not written in Java. This is a provocative title and naturally caused a lot of passionate readers comment on this thread. The thread was started in response to a posting by Nati Shalom, which seems to have been prompted by a similar posting at highscalability.com. While I don’t disagree with the analysis presented in Nati Shalom’s post, I don’t think Java, the programming language, or JEE is to be blamed. The key driver behind most successful large scale web sites is that they are designed to perform, often taking the most unorthodox approaches possible towards scalability, and most JEE developers have neither the incentive nor the experience to make their sites scalable. For the record, I have worked for over seven years for BEA Systems until I joined Yahoo about a month ago, and have great respect and admiration for BEA’s middleware stack. Sessions: HTTP sessions don’t, in general, scale. One last point.

Why most large-scale Web sites are not written in Java I think that looks pretty nice. But it looks nothing like what I have to work with. I'm in 'how do I get from there to here' hell. I'm in the actions, then the jsps, then struts config and back to the jsp to look at some javascript and then to some beans etc. I feel that pain. One thing to note too is that there is not a lick of Java in that code. You're absolutely correct on both counts. And yes that does give me some hope. Our ca. 2000 Struts project no doubt looks exactly like yours. This can become, simply using JSTL: A simple step eliminating little Java bits. The tag prefixes the path for you, and the "border='0'" comes "for free" now.

Solving Problems With Spring Boot I first became aware of Spring Boot early in 2013. At the time, we were rebuilding Spring's website and decided to write a collection of guides that could be consumed during a single lunch break (at spring.io/guides). Realizing how much code Spring Boot saved us from writing (and explaining to readers) thanks to its auto-configuration feature, we embraced it fully. In fact, it led me to write several patches for Spring Boot to help with several of the guides I was writing. Another of Spring Boot's amazing features was its support for properties. Another golden feature is Spring Boot's library of built-in actuators. Every feature Spring Boot provides is incredibly appealing to quickly moving into coding features and deploying them into real apps. I was speaking with a colleague a couple months ago and discovered he was helping a local school to setup a computer science workshop. So taking all this into account, I started writing my proposal earlier this year about Spring Boot.

Phil Libin of Evernote: The Surprise 3 A.M. Email That Saved His $200M Company | The New OPEN Forum How a mysterious man who loved Evernote ended up saving the company on the night before owner Phil Libin was going to shut it down. February 01, 2013 It was in the middle of a chilly night in October 2008 when Phil Libin started feeling, as he describes it, “like an adult.” His company, Evernote, an online service enabling users to create, catalog and search for notes (in the form of video, audio and text), was going under. Investors were fleeing and Libin had a measly three weeks worth of cash to his name. As he walked into his bedroom that night, he resigned himself to the fact that the following morning he would shut down his company and lay off all 20 employees. Less than 40 minutes later, Libin was on a Skype call with the mysterious man, and just two weeks later, was on the receiving end of a $500,000 wire transfer. Today, Evernote is one of world’s fastest growing Internet companies. Phil Libin: Nope. evernote.com KM: So what happened after that? PL: It’s very tough. PL: Yep.

How Can Your Business Benefit From Pinterest’s New Analytics Tool? Last week Pinterest finally launched what many business users on the image-sharing network have been crying out for – its brand new web analytics. Why all the excitement? Well up until now, Pinterest users have found it difficult to gain meaningful insights into how their audience was engaging with their content. Whilst you could check new followers, likes, repins and comments via the activity feed, unless you were using third-party solutions like the excellent Curalate, much of the tracking available within Pinterest involved guesswork and assumptions. Although they’re nowhere near the sophistication of Facebook Insights, Pinterest’s analytics are definitely a step in the right direction for bloggers, businesses and organizations wanting to find out what people are pinning from their websites. Let’s take a look at how you can make this new tool work for you. Get Your Page Ready For Pinterest Analytics So let’s get started. #1 Verify Your Page #2 Activate Pinterest’s New Look Key Takeaways

How can I use Python for large scale development

Related: