background preloader

How FriendFeed uses MySQL to store schema-less data - Bret Taylo

How FriendFeed uses MySQL to store schema-less data - Bret Taylo
Background We use MySQL for storing all of the data in FriendFeed. Our database has grown a lot as our user base has grown. We now store over 250 million entries and a bunch of other data, from comments and "likes" to friend lists. As our database has grown, we have tried to iteratively deal with the scaling issues that come with rapid growth. We did the typical things, like using read slaves and memcache to increase read throughput and sharding our database to improve write throughput. In particular, making schema changes or adding indexes to a database with more than 10 - 20 million rows completely locks the database for hours at a time. Lots of projects exist designed to tackle the problem storing data with flexible schemas and building new indexes on the fly (e.g., CouchDB). After some deliberation, we decided to implement a "schema-less" storage system on top of MySQL rather than use a completely new storage system. Overview Details Indexes are stored in separate tables. . Performance

Rhino serialization - MDC Beginning with Rhino 1.5 Release 3 it is possible to serialize JavaScript objects, including functions and scripts. However, serialization of code in compilation mode has some significant limitations. Serialization provides a way to save the state of an object and write it out to a file or send it across a network connection. Simple serialization example The Rhino shell has two new top-level functions, serialize and deserialize. They're intended mainly as examples of the use of serialization: $ java org.mozilla.javascript.tools.shell.Mainjs> function f() { return 3; }js> serialize(f, "f.ser")js> quit() $ java org.mozilla.javascript.tools.shell.Mainjs> f = deserialize("f.ser")function f() { return 3;}js> f()3js> Here we see a simple case of a function being serialized to a file and then read into a new instance of Rhino and called. Rhino serialization APIs Two new classes, ScriptableOutputStream and ScriptableInputStream, were introduced to handle serialization of Rhino classes.

Blog » Jaxer + Persevere via Dojo’s JsonRestStore With the increasing popularity of JavaScript, and rising usage of JavaScript in the browser, it is becoming very effective to use JavaScript on the server to have a single end-to-end language for both client and server-side development. Options for server-side JavaScript are also increasing. Jaxer is a new server that uses server-side JavaScript for HTML generation, using the standard browser DOM model (it is actually running Mozilla’s engine on the server) as the API. Getting started with Persevere and Jaxer is simple, just download, unzip, and start Persevere with java -jar start.jar, and Jaxer with the StartServers script. Or the equivalent object creation can be done with an HTTP/REST request: Next, you need to add Dojo to Jaxer’s web directory at /jaxer/aptana. Now we can easily query the table, we will do an empty query to get all the objects in the Contact table: And now we have the result set from the database and we can generate HTML with it:

Scripting Java - MDC This article shows how to use Rhino to reach beyond JavaScript into Java. Scripting Java has many uses. It allows us to write powerful scripts quickly by making use of the many Java libraries available. We can test Java classes by writing scripts. We can also aid our Java development by using scripting for exploratory programming. Note that the ECMA standard doesn't cover communication with Java (or with any external object system for that matter). Accessing Java Packages and Classes Every piece of Java code is part of a class. Rhino defines a top-level variable named Packages. js> Packages.java [JavaPackage java] As a handy shortcut, Rhino defines a top-level variable java that is equivalent to Packages.java. js> java [JavaPackage java] We can access Java classes simply by stepping down the package hierarchy: js> java.io.File [JavaClass java.io.File] If your scripts access a lot of different Java classes it can get awkward to use the full package name of the class every time.

HTTP Interface (Persevere) The Persevere Server persisted data can be accessed and manipulated with a simple and intuitive JSON REST interface. You can store, retrieve, and modify persisted data using standard REST semantics with the following HTTP methods. Clients communicate through these standard HTTP methods. Note - data is transmitted by default in the application/javascript format, allowing things like dates and functions (not included in the JSON spec) to be transmitted nicely. The GET method can be used to retrieve an object by id, a value by path, or a result set by JSONQuery. GET /Customer/1 {"id":"1", "name","John", To retrieve a value, simply use property path access appended to a starting id. GET /Customer/1.firstName "John" You can use more complex paths: GET /BlogPost/1.comments[0] "lastName":"customer", "phoneNumber":"a phone?" "address":"somewhere", "created":"2007-10-23T14:40:18Z", "firstName":"first", "comment":"this is great" You can also use JSONQuery queries to retrieve query result sets from the server:

Blog » When to Use Persevere: a Comparison with CouchDB and Othe Persevere and CouchDB are both new JSON-based storage systems, and have been receiving increasing focus. It may be informative to those interested in these new technologies to discuss the differences between these new database systems and what types of applications each is best suited for. While Persevere and CouchDB share an extremely similar JSON HTTP/REST interface and object structured data storage, there are important distinctions. Each of these different tools has a different target audience, and despite my obvious bias being the developer of Persevere, I hope to give at least some semi-objective differentiations between these tools. Fundamentally, CouchDB is primarily focused on being a general purpose database that is an alternative to a relational database. We will now take more in-depth look at the differences between Persevere and CouchDB (focusing on CouchDB due to it’s similarity to Persevere and its popularity). Querying Server-Side JavaScript Integration Security Schema

Slacker databases break all the old rules | Data Management - In So you've got some data to store. In the past, the answer was simple: Hook up an official database, pour the data into it, and let the machine sort everything out for you while you spend your time writing big checks to the database manufacturer. Now things aren't so cut and dry. A fresh round of exciting new tools is tacking the two letters "db" onto a pile of code that breaks with the traditional relational model. Old database administrators call them "toys" and hint at terrible dangers to come from the follies of these young whippersnappers. The whippersnappers just tune out the warnings because the new tools are good enough and fast enough for what they need. The non-relational upstarts are grabbing attention because they're willfully ignoring many of the rules that codify the hard lessons learned by the old database masters. [ For a brief look at more alternative databases, see Open source and SaaS offerings rethink the database. The first thing to go is the venerable old JOIN.

Related: