background preloader

Scaling Twitter: Making Twitter 10000 Percent Faster

Scaling Twitter: Making Twitter 10000 Percent Faster
Update 6: Some interesting changes from Twitter's Evan Weaver: everything in RAM now, database is a backup; peaks at 300 tweets/second; every tweet followed by average 126 people; vector cache of tweet IDs; row cache; fragment cache; page cache; keep separate caches; GC makes Ruby optimization resistant so went with Scala; Thrift and HTTP are used internally; 100s internal requests for every external request; rewrote MQ but kept interface the same; 3 queues are used to load balance requests; extensive A/B testing for backwards capability; switched to C memcached client for speed; optimize critical path; faster to get the cached results from the network memory than recompute them locally.Update 5: Twitter on Scala. A Conversation with Steve Jenson, Alex Payne, and Robey Pointer by Bill Venners. Twitter started as a side project and blew up fast, going from 0 to millions of page views within a few terrifying months.

your database in the cloud By Navneet Joneja, Product Manager for Google Cloud SQL Cross-posted from the Google App Engine Blog One of App Engine’s most requested features has been a simple way to develop traditional database-driven applications. You can now choose to power your App Engine applications with a familiar relational database in a fully-managed cloud environment. Google Cloud SQL brings many benefits to the App Engine community: No maintenance or administration - we manage the database for you.High reliability and availability - your data is replicated synchronously to multiple data centers. Cloud SQL is available free of charge for now, and we will publish pricing at least 30 days before charging for it. Navneet Joneja loves being at the forefront of the next generation of simple and reliable software infrastructure, the foundation on which next-generation technology is being built.

How does Twitter search work Chocolatey Gallery What can I use to profile C++ code in Linux Guerrilla Manual Online Because of the great diversity of time scales that exist in modern computer systems, it's a good idea to try and get a more intuitive feel for some of them. My first attempt at helping you to do that was in 1998, where I included the table shown at the left in The Practical Performance Analyst. I updated that as Table 3.1 in my Perl PDQ book. Here, I've rendered those quantities as a data frame in R: Subsystem nanoSeconds secondZ Rescaling SIunit 1 4 GHz CPU clock 2.50e-01 0.25 1.00 s 2 L1 cache access 5.00e-01 0.50 1.00 s 3 L2 cache access 1.25e+00 1.25 1.00 s 4 Memory bus cycle 2.00e+00 2.00 1.00 s 5 DRAM chip access 6.00e+01 60.00 1.00 min 6 Physical disk seek 3.50e+06 3500000.00 1.33 month 7 Network NFS read 3.20e+07 32000000.00 1.01 yr 8 Database SQL update 5.00e+08 500000000.00 15.84 yr 9 Magnetic tape read 5.00e+09 5000000000.00 1.58 century To get a better impression of the vast range of time scales, actual nanoseconds are rescaled to be units measured in seconds.

Type and search, how to improve the performance 19 ffmpeg commands for all needs Getting infos from a video file ffmpeg -i video.avi Turn X images to a video sequence ffmpeg -f image2 -i image%d.jpg video.mpg This command will transform all the images from the current directory (named image1.jpg, image2.jpg, etc…) to a video file named video.mpg. Turn a video to X images ffmpeg -i video.mpg image%d.jpg This command will generate the files named image1.jpg, image2.jpg, … The following image formats are also availables : PGM, PPM, PAM, PGMYUV, JPEG, GIF, PNG, TIFF, SGI. Encode a video sequence for the iPpod/iPhone ffmpeg -i source_video.avi input -acodec aac -ab 128kb -vcodec mpeg4 -b 1200kb -mbd 2 -flags +4mv+trell -aic 2 -cmp 2 -subcmp 2 -s 320x180 -title X final_video.mp4 Explanations : Source : source_video.aviAudio codec : aacAudio bitrate : 128kb/sVideo codec : mpeg4Video bitrate : 1200kb/sVideo size : 320px par 180pxGenerated video : final_video.mp4 Encode video for the PSP Extracting sound from a video, and save it as Mp3 Convert a wav file to Mp3 Convert .avi video to .mpg

网卡性能分析-Intel8257X芯片手册读后感 - 从Java到C再到Linux - CSDN博客 引:在《《OpenVPN性能》之后,我进一步阅读了硬件的解决方案,希望能得到一些思想,然后进一步的改进我的设计,由于工作的便利性和实际工作的需要,我阅读了intel的82571EB,82574L,82575等以太网芯片的datesheet的相关特性描述部分(由于我不打算亲自写驱动,因此我没有阅读寄存器以及存储器细节,更多的是我不相信自己的驱动比intel的工程师们的更高效),得到了很多感觉,以下是我的一些摘录和读后感。 一,网络应用开销 1.协议栈处理开销:分层模型各个层次的OS实现开销 2.内存拷贝开销:网卡,内核内存,用户态内存之间的拷贝 3.系统层面的开销:中断,缓存管理,系统调用 二,局部的解决方案 1.dma-针对内存拷贝,需要锁定总线,此时cpu就好像被拔除了一样(《Intel微处理器》中原话),如果网卡数据芯片没有cpu高效,除了省去了一次cpu中转之外,性能反而降低。 2.硬中断负载均衡-针对多cpu的利用率(希望多cpu全部用来处理协议栈),此带来软中断负载均衡(在硬中断cpu上触发软中断),做的不好没有效果,造成某个cpu高负载,其它cpu空闲,做的好的话,会造成基于顺序的协议包乱序,这就是cpu并行和tcp串行之间的冲突。 3.TOE-针对pci总线访问延迟过高,tcp卸载引擎,这种方案对于经常传输小包的情形来说无疑是不好的,因此在网卡中大量处理tcp协议会造成收发速率降低。 ******************* For cases where the 82575 is connected to a small number of clients, it is desirable to initiate the interrupt as soon as possible with minimum latency. ******************* 2.针对header split,82575手册上有很好的描述, This feature consists of splitting or replicating a packet’s header to a different memory space. *************** 如果不是很理解,微软的msdn上也有描述,截图如下: **************

Related: