Compiling, installing and test-running Scribe. I went to the Hadoop World conference last week and one thing I took away was how Facebook and other companies handle the problem of scalable logging within their infrastructure. The solution found by Facebook was to write their own logging server software called Scribe (more details on the FB blog). Scribe is mentioned in one of the best presentations I attended at the conference -- 'Hadoop and Hive Development at Facebook' by Dhruba Borthakur and Zheng Shao. If you look at page 4, you'll see the enormity of the situation they're facing: 4 TB of compressed data (mostly logs) handled every day, and 135 TB of compressed data scanned every day. All this goes through Scribe, so that gives me a warm fuzzy feeling that it's indeed scalable and robust. 1) Install pre-requisite packages On Ubuntu, I had to install the following packages via apt-get: g++, make, build-essential, flex, bison, libtool, mono-gmcs, libevent-dev. - untar it, then cd into the boost directory and run: $ . 4) Install Scribe.
[Thrift] Handling failover and high availability with thriftserv. [Thrift] Handling failover and high availability with thriftservices (oh, and load balancing too) Previous message: [Thrift] Handling failover and high availability with thrift services (oh, and load balancing too) Next message: [Thrift] Perl package patch + lex error fix Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] Hi guys, First of all, we're thrilled to hear you saying that some of this stuff doesn't belong in Thrift. We totally agree. Keeping the core Thrift library super lightweight is one of the main goals, while also designing to make it easy to extend, build support libraries for, or add wrappers/tools for more complicated tasks.
Presentation Summary “High Performance at Massive Scale: Lessons. Recently, we were fortunate to host Jeff Rothschild, the Vice President of Technology at Facebook, for a visit for the CNS lecture series. Jeff’s talk, “High Performance at Massive Scale: Lessons Learned at Facebook” was highly detailed, providing real insights into the Facebook architecture. Jeff spoke to a packed house of faculty, staff, and students interested in the technology and research challenges associated with running and Internet service at scale. The talk is archived here as part of the CNS lecture series. I encourage you to check it out; below are my notes on the presentation. Site Statistics: Photo sharing on Facebook: Facebook stores 20 billion photos in 4 resolutions2-3 billion new photos uploaded every monthOriginally provisioned photo storage for 6 months, but blew through available storage in 1.5 weeks.Facebook serves 600k photos/second –> serving them is more difficult than storing them.
Scaling photos, first the easy way: Scaling photos, 2nd generation: Site Architecture. Facebook engineering interview. Earlier today (or should I say yesterday) I attended Facebook’s Seattle Engineering Road Show which was a part technical talk and part recruiting event where Mike Shroepfer and a number of other Facebook engineers gave a fairly deep technical talk about the technologies used by Facebook. Below are my notes on the presentation and ensuing Q&A session. One thing I found interesting about the talk was how similar their architecture for the platform that powers the Facebook news feed is to what my team has built for powering the Windows Live What's New feed. Before the presentation started there were a bunch of slides that carried some interesting and sometimes impressive stats about Facebook including Mike Shreopfer started his talk with a number of impressive statistics. Scaling Facebook has proven more challenging than scaling a traditional website.
The most famous example of tackling this challenge is the Facebook news feed. Now Playing: Yo Gotti - 5 Star (Remix) [feat. Scaling Chat Engineering @ Facebook's Notes. Scale at Facebook. Facebook: Moving Fast at Scale. A Day in the Life of Facebook Operations: Velocity 2010, Web Per. Facebook is now the #2 global website, responsible for billions of photos, conversations, and interactions between people all around the world running on top of tens of thousands of servers spread across multiple geographically-separated datacenters. When problems arise in the infrastructure behind the scenes it directly impacts the ability of people to connect and share with those they care about around the World. Facebook’s Technical Operations team has to balance this need for constant availability with a fast-moving and experimental engineering culture. We release code every day. Additionally, we are supporting exponential user growth while still managing an exceptionally high radio of users per employee within engineering and operations.