background preloader

Twitter Architecture

Twitter Architecture
Toy solutions solving Twitter’s “problems” are a favorite scalability trope. Everybody has this idea that Twitter is easy. With a little architectural hand waving we have a scalable Twitter, just that simple. Well, it’s not that simple as Raffi Krikorian, VP of Engineering at Twitter, describes in his superb and very detailed presentation on Timelines at Scale. If you want to know how Twitter works - then start here. It happened gradually so you may have missed it, but Twitter has grown up. Twitter now has 150M world wide active users, handles 300K QPS to generate timelines, and a firehose that churns out 22 MB/sec. 400 million tweets a day flow through the system and it can take up to 5 minutes for a tweet to flow from Lady Gaga’s fingers to her 31 million followers. A couple of points stood out: Twitter no longer wants to be a web app. How does Twitter work? The Challenge Groups The Platform Services group is responsible for the core scalable infrastructure of Twitter. Push Me Pull Me

Related:  Server-sideSoftware Architecture

Scaling Pinterest - From 0 to 10s of Billions of Page Views a Month in Two Years Pinterest has been riding an exponential growth curve, doubling every month and half. They’ve gone from 0 to 10s of billions of page views a month in two years, from 2 founders and one engineer to over 40 engineers, from one little MySQL server to 180 Web Engines, 240 API Engines, 88 MySQL DBs (cc2.8xlarge) + 1 slave each, 110 Redis Instances, and 200 Memcache Instances. Stunning growth. ZeroMQ ZeroMQ is a messaging library, which allows you to design a complex communication system without much effort. It has been wrestling with how to effectively describe itself in the recent years. In the beginning it was introduced as ‘messaging middleware’ later they moved to ‘TCP on steroids’ and right now it is a ‘new layer on the networking stack’. I had some trouble understanding ZeroMQ at first and really had to reset my brain. First of all, it is not a complete messaging system such as RabbitMQ or ActiveMQ. I know the guys of Linden Research compared them, but it is apples and oranges.

Twitter’s Root Injustice — - meta - How To Make Twitter More Fair and Competitive The story of my experience on Twitter is the story that is celebrated—but rarely the case for most people. When I joined Twitter in 2009 it was like showing up to a half-settled frontier town. I could talk with people who would normally never talk to me, I could gain a following in fields where I was years younger than most, and I could do this all from a laptop in my underwear. Twitter was magical in 2009 because the platform was getting a daily flood of users of all stripes eager to find accounts to follow. We’ll never be able to get back to that era of Twitter, but there are a ton of things Twitter can do to help users.

How we built our Real-time Analytics Platform » MaxCDN Blog With the recent release of our Analytics Platform, we would like to give you a behind the scenes look of how we built it. The amount of log data our worldwide network produces is staggering. Every day, our servers generate over 8 terabytes of raw log data (before compression). During our peak times, our servers are generating over 200,000 requests per second. As we grow, we will see this number continue to climb. Software Architecture AntiPatterns Architecture AntiPatterns focus on the system-level and enterprise-level structure of applications and components. Although the engineering discipline of software architecture is relatively immature, what has been determined repeatedly by software research and experience is the overarching importance of architecture in software development: Good architecture is a critical factor in the success of the system development.Architecture-driven software development is the most effective approach to building systems. Architecture-driven approaches are superior to requirements-driven, document-driven, and methodology-driven approaches.

Twitter News Consumers: Young, Mobile and Educated Nearly one-in-ten U.S. adults (8%) get news through Twitter, according to a new report by the Pew Research Center, in collaboration with the John S. and James L. Knight Foundation. Compared with the 30% of Americans who get news on Facebook, Twitter news consumers stand out as younger, more mobile and more educated. In addition, a separate Pew Research analysis of conversations on Twitter around major news events reveals three common characteristics: much of what gets posted centers on passing along breaking news; sentiments shift considerably over time; and however passionate, the conversations do not necessarily track with public opinion. This two-part report is based first on a survey of more than 5,000 U.S. adults (including 736 Twitter users and 3,268 Facebook users) and, second, on an analysis of Twitter conversations surrounding major news events which spanned nearly three years.

Solving Problems With Spring Boot I first became aware of Spring Boot early in 2013. At the time, we were rebuilding Spring's website and decided to write a collection of guides that could be consumed during a single lunch break (at Realizing how much code Spring Boot saved us from writing (and explaining to readers) thanks to its auto-configuration feature, we embraced it fully. In fact, it led me to write several patches for Spring Boot to help with several of the guides I was writing. Discovering that boiler plate Spring code was unnecessary was incredibly exciting and very effective. Another of Spring Boot's amazing features was its support for properties.

Wikipedia on HHVM « HHVM Posted on January 6, 2015 by Brett Simmers If you’ve been watching our GitHub wiki, following us on Twitter, or reading the wikitech-l mailing list, you’ve probably known for a while that Wikipedia has been transitioning to HHVM. This has been a long process involving lots of work from many different people, and as of a few weeks ago, all non-cached API and web traffic is being served by HHVM. This blog post from the Wikimedia Foundation contains some details about the switch, as does their page about HHVM. I spent four weeks in July and August of 2014 working at the Wikimedia Foundation office in San Francisco to help them out with some final migration issues.

Twitter's New Design Looks A Lot Like Facebook, And That's A Good Thing After months of testing, Twitter has announced a new design that makes the popular micro-blogging service look a lot more like Facebook. But don't let that fool you. Twitter's as dedicated to being a digital stream of consciousness as ever. Node.js in Production When running a node application in production, you need to keep stability, performance, security, and maintainability in mind. Outlined here is what I think are the best practices for putting node.js into production. By the end of this guide, this setup will include 3 servers: a load balancer (lb) and 2 app servers (app1 and app2). The load balancer will health check and balance traffic between the servers.

Software Architecture Review Guidelines Introduction Reviews are an opportunity for others to eyeball your documents/design/code/software architecture and for you to inspect others' work. They facilitate knowledge interchange. But their primary goal is to increase software quality.

In Defense of Getting Personal on Twitter - The Digital Campus 2014 By Alice E. Marwick Keep it professional. That’s what we tell high-school and college students about their Facebook and Twitter accounts. We explain that employers can and will mine your personal social media for data, and that what you say online can follow you forever. Node.js w/1M concurrent connections! I’ve decided to ramp up the Node.js experiments, and pass the 1 million concurrent connections milestone. It worked, using a swarm of 500 Amazon EC2 test clients, each establishing ~2000 active long-poll COMET connections to a single 15GB rackspace cloud server. This isn’t landing the mars rover, or curing cancer.