background preloader

Distributed computing

Facebook Twitter

Distributed Systems and the End of the API. Akka Notes - Actor Messaging - 1. From the introductory first part of the Akka Notes, we saw a bird’s eye view of Actors in the Akka Toolkit. In this second part of the Akka Notes, we’ll look at the messaging part of Actors. As for the example, we would use the same Student-Teacher example that we discussed earlier. In this first part of Actor Messaging, we’ll create the Teacher Actor and instead of the Student Actor, we’ll use a main program called StudentSimulatorApp. Revisiting Student-Teacher in Detail Let’s for now consider the message sent by the StudentSimulatorApp to the TeacherActor alone. When I say StudentSimulatorApp, I just mean a normal main program. The picture conveys this : (if the terms are overwhelming, don’t worry, we’ll go through them in detail) Student creates something called an ActorSystemIt uses the ActorSystem to create something called as ActorRef. Like I said, don’t worry about it. The StudentSimulatorApp program As we understand from the picture, the StudentSimulatorApp Creating an ActorSystem Code.

Akka Notes - Actor Logging and Testing. In the first two parts (one, two), we briefly talked about Actors and how messaging works. In this part, let’s look at fixing up Logging and Testing our TeacherActor. Recap This is how our Actor from the previous part looked like : Logging Akka with SLF4J You notice that in the code we are printing the quoteResponse to the standard output which you would obviously agree is a bad idea. 1. Akka provides a nice little trait called ActorLogging to achieve it. A small detour here : Internally, when we log a message, the logging methods in the ActorLogging (eventually) publish the log message to an EventStream. EventStream and Logging EventStream behaves just like a message broker to which we could publish and receive messages. In case of logging messages, all log messages would be published to the EventStream. So, that’s the reason when we try to kick off the StudentSimulatorApp, we see the log message written to the console. That said, EventStream isn’t suited only for logging. 2. 3.

Testing Akka. Akka Notes - Actor Messaging - Request and Response - 3. Last time when we saw Actor messaging, we saw how fire-n-forget messages are sent (Meaning, we just send a message to the Actor but don’t expect a response from the Actor). Technically, we fire messages to Actors for its side-effects ALL THE TIME. It is by design. Other than not responding, the target Actor could ALSO do the following with that message: Send a response back to the sender (in our case, the TeacherActor would respond with a quote back to the StudentActor ORForward a response back to some other Actor who might be the intended audience which in turn might respond/forward/have a side-effect.

Request & Response In this write-up, we’ll be focussing only on Point 1 – the request-response cycle. The picture conveys what we are trying to achieve this time. We’ll also cook up a testcase to verify it. Let’s look at these 4 points in detail now : 1. By now, you would have guessed what would the DriverApp do. Initialize the ActorSystem Create the TeacherActor Create the StudentActor Lastly, Google Releases gRPC, a HTTP/2 RPC Framework for Microservices. Google has opened sourced gRPC, a RPC framework used internally to connect cloud microservices. gRPC comes with support for 10 languages, making it attractive for creating back-end cloud services for mobile applications. gRPC is a language and platform-neutral RPC system developed and used internally by Google in many areas including cloud computing services, and they are going to expose most of their public APIs through such endpoints.

Google considers gRPC a “bandwidth and CPU efficient, low latency way to create massively distributed systems that span data centers, as well as power mobile apps, real-time communications, IoT devices and APIs.” While it can be used in many scenarios, gRPC is targeted primarily at microservices. A service interface and the data types it handles is defined using the Protocol Buffers IDL, and with the help of compilers one can generate client and server stubs in 10 languages: C, C++, C#, Go, Java, Node.js, Objective-C, PHP, Python, Ruby. Scaling Apache Solr. First time accepted submitter sobczakt writes We live in a world flooded by data and information and all realize that if we can't find what we're looking for (e.g. a specific document), there's no benefit from all these data stores.

When your data sets become enormous or your systems need to process thousands of messages a second, you need to an environment that is efficient, tunable and ready for scaling. We all need well-designed search technology. A few days ago, a book called Scaling Apache Solr landed on my desk. The author, Hrishikesh Vijay Karambelkar, has written an extremely useful guide to one of the most popular open-source search platforms, Apache Solr. Solr is a full-text, standalone, Java search engine based on Lucene, another successful Apache project. Karambelkar is an enterprise architect with a long history in both commercial products and open source technology. The book is divided into 10 chapters. Chapter 8, dedicated to optimization, drew my attention. Apache Helix: A framework for Distributed System Development. This post originally appeared as a contributed piece on The New Stack.

Distributed data systems are used in a variety of settings like online serving, offline analytics, data transport, and search, among other use cases. These start off with a single node solution that provides the core functionality e.g. it can be a database, messaging, search index etc. Building from a single node, requires not just a distributed system but also the different requirements that go with it. With these issues related to scaling in mind, we built Helix, a generic framework for developing distributed systems. Why Helix For a long time, a single node solution was sufficient for most use cases but with the advent of big data thats no longer true. Most applications need to run in a distributed setting and with distributed computing come challenges of scalability, partition and fault-tolerance. Adding these capabilities is non trivial, error prone and time consuming.

Distributed System Challenges Requirements.