Scalability

Facebook Twitter

Amazon's Dynamo. In two weeks we’ll present a paper on the Dynamo technology at SOSP, the prestigious biannual Operating Systems conference.

Amazon's Dynamo

Dynamo is internal technology developed at Amazon to address the need for an incrementally scalable, highly-available key-value storage system. The technology is designed to give its users the ability to trade-off cost, consistency, durability and performance, while maintaining high-availability. Let me emphasize the internal technology part before it gets misunderstood: Dynamo is not directly exposed externally as a web service; however, Dynamo and similar Amazon technologies are used to power parts of our Amazon Web Services, such as S3.

We submitted the technology for publication in SOSP because many of the techniques used in Dynamo originate in the operating systems and distributed systems research of the past years; DHTs, consistent hashing, versioning, vector clocks, quorum, anti-entropy based recovery, etc. Java NIO. Comme nous l’avons déjà évoqué sur le blog, à l’occasion du challenge USI 2011, nous nous sommes intéressés à différents serveurs et framework web NIO en Java.

Java NIO

Le principe était simple en mettant à plat la spécification du challenge, nous avons identifié quelques besoins techniques : Une solution pour le marshalling JSONUn serveur web NIO supportant le long pollingUne solution pour la persistence et le partage des données Notre démarche a été de réaliser des POCs implémentant la création des utilisateurs et le long polling pour retenir la meilleure solution. La solution devait être simple et rapide à implémenter, et tenir une charge conséquente en la testant à l’aide de ab l’outil de benchmark Apache et de la librairie Async Http Client.

Pour le JSON, nous nous sommes tous rapidement mis d’accord sur l’utilisation de la librairie Jackson. Building Scalable Systems: an Asynchronous Approach. Bio Theo Schlossnagle is the author of Scalable Internet Architectures (SAMS) and a frequent speaker at worldwide IT conferences.

Building Scalable Systems: an Asynchronous Approach

He was the Principal Architect of the Momentum MTA, a new email system. Theo is a member of the IEEE and a senior member of the ACM. He serves on the editorial board of the ACM’s Queue Magazine. You can follow Theo on Twitter as @postwait. The LMAX Architecture. LMAX is a new retail financial trading platform.

The LMAX Architecture

As a result it has to process many trades with low latency. The system is built on the JVM platform and centers on a Business Logic Processor that can handle 6 million orders per second on a single thread. The Business Logic Processor runs entirely in-memory using event sourcing. Concurrent Programming Using The Disruptor. Bio Trisha Gee is a developer at LMAX, the London Multi Asset eXchange, trying to get her head around low-latency, high performance coding whilst also keeping her fingers in the other pies LMAX has to offer, such as continuous delivery and agile.

Concurrent Programming Using The Disruptor

Trisha is heavily involved in the London Java Community and the Graduate Development Community. Software is changing the world; QCon aims to empower software development by facilitating the spread of knowledge and innovation in the enterprise software development community; to achieve this, QCon is organized as a practitioner-driven conference designed for people influencing innovation in their teams: team leads, architects, project managers, engineering directors. Facebook's Realtime Analytics System. Recently, I was reading Todd Hoff's write-up on FaceBook real time analytics system.

Facebook's Realtime Analytics System

As usual, Todd did an excellent job in summarizing this video from Engineering Manager at Facebook Alex Himel, Engineering Manager at Facebook. In this first post, I’d like to summarize the case study, and consider some things that weren't mentioned in the summaries. High Scalability.