There are many products and projects out there that allow you to stream data to clients from a server, and most letting you send data back to a server too. There are various approaches that have been taken to achieve this which will be explored in this article.
In a followup to this post I’ll explore some of the advantages and disadvantages of the methods discussed here further and talk a bit about domain driven design.
The term Comet came along a few years ago as a general term to describe the case where the client is a web browser and there are no visible page reloads. Many servers in this space support other client side technologies too, but the focus is often web browser clients.
When designing a product in this category there are various decisions to make. Many commercial products have target audiences in mind at this stage, whereas some projects are trying to solve a technical problem and the real world use cases will come later. In both cases technical decisions need to be made regarding how the product will look to the user (the developer) and how it will be implemented under the covers.
The complexity of the client API is a key aspect. There is a big trade off here between how much work a client coder has to do and how open what he can do is. This isn’t a simple scale though and the trick is to have something that is very easy to use without losing out on any flexibility in what can be achieved. A socket like API leaves a reasonable amount of work for the client coder, but also means he is free to implement almost anything. Conversely, many products in this space provide a publish/subscribe like API, which gives the client coder some basic building blocks that a good proportion of applications are likely to find helpful.
Publish/Subscribe
The use of the pub/sub paradigm has become very common in this space. It is a natural fit. The client chooses what it wants to see on its screen and subscribes to it from the server. Clients, or other server components, publish data and the server pushes that data to all subscribed clients. Many applications fit into this template, and some that do not can often still make use of the pub/sub API in a simple way without too much of an overhead.
Pub/sub may be common, however it is implemented differently by different products. JMS like systems can be fairly static, you configure topics/destinations and these are used for a class of message or a specific application. Other systems are more dynamic, and to use an example from the finance world, the topic would represent a stock or other financial instrument and are created dynamically as they are needed. Some products that use the first, more static, method will have a sub topic which is more suited for use as the stock name.
Another way that pub/sub implementations can differ is in what happens when you first subscribe. In pure pub/sub, which may be based on multicast technology, when you subscribe you will get a message the next time there is a message on that topic. This is not always that useful, to use an example from the finance world again, when you subscribe to a stock you want to know its current value along with subsequent changes. Some older pub/sub systems add this capability in, but the initial value comes from a different place, introducing a nasty race condition.
Messages
The format of the messages, both on the wire and how they are presented to the client coder, can differ a lot between products. Not all products will present what they do as messaging, but at some level that is what they are doing, something is being sent from the server to the client or vice versa.
Sockets
The most basic method here is a socket-like API. The client coder is left to implement a protocol using this API. This gives maximum flexibility at the expense of possible complexity of the client coders implementation. With a browser client, a Socket-like API (see WebSocket) does a bit more for you compared to a true socket, it will most likely present you with whole messages, which avoids tricky code to handle partial reads etc.
Using sockets means you do not have any features like pub/sub or structured messages, it all has to be implemented by the client coder.
This approach is good for the hardcore coders who want maximum flexibility. It can also more easily integrate with existing backend servers that have differing protocols.
Simple Pub/Sub
A simple pub/sub approach with basic message structures can be very useful and solve a lot of use cases. Pub/sub means your messages have a subject (or topic) to identify it and a message body. The message body can vary between products, field/value pairs, structured data types, opaque data types etc.
This is an area that many products fall into and it’s the message body and the API that differentiates them at this level.
Field/Value pairs
This is a very simple, but powerful, structure. It is basically a hashtable or a map structure, often ordered. Most things can be represented in this format, but sometimes more complex structures are needed to avoid extra encoding being used by the client coder.
Structured data types
A reasonably common approach here is to use a format such as JSON for the message body. JSON allows simple data structures and objects to be represented in a textual human readable format. This gives a bit more flexibility over a field/value pair message type.
Object Conversion
Another approach is to send objects as the message body. This is similar to the JSON approach in many ways, but the object is converted from one languages representation of an object into a serialised form and then back again into an object in the language of the receiving end. For example this could be a Java object being converted to a wire format and then being converted to a Javascript object or an ActionScript object. Technologies like Flex/LCDS use this with AMF wire protocol to good effect, which can reduce bandwidth and processing compared to sending complex objects a formats such as XML.
Object Remoting
Object conversion is most likely used within a pub/sub like system, the object being the message body, whereas object remoting is more like conventional object oriented programming.
Object remoting tries to abstract away the fact that a client is talking to a server over a network. The client has an object that is a proxy for an object on the server (or vice versa) and allows method calls to be made on the remote objects. This means once setup, the client code can be written much like a normal application, making method calls on objects as if they were local. If there are return values for the method calls then there can be issues that complicate the apparent simplicity, as you will have to block for the call to return, or have a callback mechanism.
My followup article will cover why some of these approaches can be improved upon if your product is targeting a specific domain and how this can lead to faster time to market and a better final solution.