Comet server message sizes, Bandwidth considerations

One of the questions I brought up on a previous blog – Comet servers for a Single-Dealer Platform – was that of bandwidth. The main thing that affects bandwidth that a Comet server has some control over is message size. At the time of that blog I hadn’t looked into this issue with all the servers in much detail, so I decided to dive a little deeper and look at the protocols of some of the Comet servers written about on that blog.

A question that a customer once asked was about message sizes, but the question was wrong, they asked what performance Liberator could handle with 1K messages – they wanted to compare Liberator to another Comet server sending 1K messages. I pointed out that they should let us know what the payload they want to send is and we can see the size of the message in our protocol, and they should do the same with the other Comet server. Maybe we could represent a much bigger payload in a 1K message than the other server could.

I have looked at four servers for this blog – Liberator, Lightstreamer, Adobe LCDS and my-Channels Nirvana. They all take slightly different approaches, but can all be used to achieve similar results. My test case is a single message payload, but I am showing the structure of the messages and will comment on how different payloads may be represented better or worse in the different servers.

The payload I am looking at is a fairly typical financial data update. To help highlight a few problems I am looking at what is often called an ‘image’ and an ‘update’. An image is an initial update, containing all the data about the instrument, the update is the subsequent updates where only a subset of all the fields will be updating.

The image is:

Field1123456
Field2234567
Time11274181389019
Low100000
High200000
FullNameCompany Name Ltd

The update is:

Field1123459
Field2234569
Time11274181389999

These may seem like small messages, but they are fairly typical, and it should be easy to see how a few more fields will affect the message sizes. Lets first look at the summary of results, and if you are interested, further down is the detail of the messages.

The table shows the size of the message in bytes that each server will send. For LCDS and Nirvana there are two entries as they implement different protocols with quite different message sizes. I assume anyone using LCDS would be using the binary AMF protocol, you get to choose. With Nirvana, if you want a native web client you have to use the web protocol, but the protocol available to other technologies has far smaller message sizes.

ServerProtocolImage Message SizeUpdate Message Size
LiberatorWeb10164
LiberatorOther8043
LightstreamerWeb9567
LightstreamerOther6436
NirvanaNHP (Web clients)760675
NirvanaNSP (Java/Other clients)154102
LCDSAMF binary403403
LCDSAMF XML937937

Lets now look at the actual message structure and how they might differ with other payloads.

Liberator

Liberator is sending a script tag wrapped Javascript function call, with a single argument containing its own protocol. The messages do not contain the subject (or channel), and field names are mapped onto single characters (for the first 64 fields, then 2 characters etc..) This is quite compact with not much overhead. As more fields or larger field values are added there is little overhead.

Web Image message = 101 bytes

<script>a("7O5W0001 a=123456 b=234567 c=1274181389019 d=100000 e=200000 f=Company+Name+Ltd")</script>

Web Update message = 64 bytes

<script>a("6c5W0002 a=123459 b=234569 c=1274181389999")</script>

Other APIs such as Java and .Net don’t need the script wrapper, which makes the messages that much smaller.

Other Image message = 80 bytes

7O5W0001 a=123456 b=234567 c=1274181389019 d=100000 e=200000 f=Company+Name+Ltd\n

Other Update message = 43 bytes

6c5W0002 a=123459 b=234569 c=1274181389999\n

Lightstreamer

Lightstreamer is similar to Liberator in that it is also sending a single Javascript function call. Again, no subject name needed, and field names are not used at all. The fields are known by the subscriber, so the order the values come back in is significant – this has the benefit that it doesn’t need to send field names, but it does mean that fields that have not updated still need to be sent, albeit as an empty string. This is illustrated by the three empty strings at the end of the update message. Worst case scenario where a subscription to lots of fields where not many updated very often would have a bit of an overhead, but it is not that significant.

Web Image message = 95 bytes

<script>z(0,1,"123456","234567","1274181389019","100000","200000", "Company Name Ltd");</script>

Web Update message = 67 bytes

<script>d(0,1,"123459","234569","1274181389999","","","");</script>

For non web APIs the messages are again smaller.

Other Image update = 64 bytes

0,1|123456|234567|1274181389019|100000|200000|Company Name Ltd\r\n

Other Update message = 36 bytes

0,1|123459|234569|1274181389999|||\r\n

Nirvana

Nirvana’s NHP protocol used by browser Javascript clients is quite verbose. Once again it is Javascript being streamed directly, but it is not that optimised for size. The channel name is sent, twice, and field names are sent both as part of the map and as a list. There is also some extra data in there, which may be useful to some of the features of Nirvana, but seems like quite an overhead.

Note that this is a message using the properties style message (the ‘fprops’ section), it is also possible to send an opaque message (or a combination of the two) – an opaque message uses the ‘badata’ part of the message and would then not need all the fprops information – but you would have to pack some of that into your opaque message.

Since you can send what data/fields you want, the update message is smaller, not containing anything about the fields that haven’t updated.
With much larger payloads, the overhead will become less significant.
The messages below have been formatted a bit for easier readability, adding line breaks, the sizes reflect the actual message sizes though.

Image message = 760 bytes

<script>
 try{
 window.parent.handleNVLLastEID("/PATH/TO/CHANNEL1","28");
 window.parent.handleNVLEvents({ eid : 29,
 cname : "/PATH/TO/CHANNEL1",
 tag : "notag",
 badata : "",
 hasData : true,
 hasDictionary : true,
 'fprops': {
 'Field1':"123456",
 'Field2':"234567",
 'Time1':1274181389019,
 'Low':"100000",
 'High':"200000",
 'FullName':"Company Name Ltd",
 'nrvpub.time':1274187070518,
 'nrvpub.host':"host.domain.com",
 'nrvpub.name':"username",
 'JMSMessageID':"My Message Id",
 'JMSXUserID':"My User id",
 'JMSDeliveryMode':"NON_PERSISTENT",
 nrvkeys : ['Field1','Field2','Time1','Low','High',
 'FullName','nrvpub.time','nrvpub.host','nrvpub.name',
 'JMSMessageID','JMSXUserID','JMSDeliveryMode'],
 nrvprops : "",
 nrvproparrays : "" }
 },
 "eventHandlerCallbackFunc");
 }catch(Exception){}
 </script>

Update message = 675 bytes

 <script>
 try{
 window.parent.handleNVLLastEID("/PATH/TO/CHANNEL1","28");
 window.parent.handleNVLEvents({ eid : 29,
 cname : "/PATH/TO/CHANNEL1",
 tag : "notag",
 badata : "",
 hasData : true,
 hasDictionary : true,
 'fprops': {
 'Field1':"123459",
 'Field2':"234569",
 'Time1':1274181389999,
 'nrvpub.time':1274187070518,
 'nrvpub.host':"host.domain.com",
 'nrvpub.name':"username",
 'JMSMessageID':"My Message Id",
 'JMSXUserID':"My User id",
 'JMSDeliveryMode':"NON_PERSISTENT",
 nrvkeys : ['Field1','Field2','Time1',
 'nrvpub.time','nrvpub.host','nrvpub.name',
 'JMSMessageID','JMSXUserID','JMSDeliveryMode'],
 nrvprops : "",
 nrvproparrays : "" }
 },
 "eventHandlerCallbackFunc");
 }catch(Exception){}
 </script>

Nirvana’s NSP protocol is used by the Java API (and other Enterprise APIs), this is a binary protocol and much smaller in message size. Field names are sent but channel names are not.

Image message = 154 bytes

..............Field1..123456.Field2..234567.Time1......J.
 Low..100000.High..200000.FullName..Company Name Ltd.........J.
 .......host.domain.com.username....

Update message = 102 bytes

..............Field1..123459.Field2..234569.Time1......J.
 ........J........host.domain.com.username....

Adobe LCDS

LCDS sends serialised representations of objects, which means it sends all the fields whether they have changed or not. You could probably construct objects to only contain changed fields, but it doesn’t play to the strengths of the intended ease of use of sending objects and object remoting.
The XML version of the message is very verbose, but I don’t think there is a reason to use it, so it is only here to show the data more clearly than the binary version of the message. Class names, channel names, field names, XML tags, they are all sent and makes the message very large. Again, line breaks and indentation have been added for readability.

AMF XML message = 937 bytes

<object type="flex.messaging.messages.AsyncMessage">
 <traits>
 <string>destination</string>
 <string>headers</string>
 <string>correlationId</string>
 <string>messageId</string>
 <string>timestamp</string>
 <string>clientId</string>
 <string>timeToLive</string>
 <string>body</string>
 </traits>
 <string>market-data-feed</string>
 <object>
 <traits>
 <string>DSSubtopic</string>
 </traits>
 <string>PATH.TO.INSTRUMENT</string>
 </object>
 <null/>
 <string>DCE35EDA-AF0F-B3B5-3ECC-DA2D0F7BE56B</string>
 <double>1.243526266655E12</double>
 <string>DCE25D41-DC0A-6A55-8F54-E7E2C34D1B07</string>
 <double>0.0</double>
 <object type="flex.samples.marketdata.Stock">
 <traits>
 <string>Field1</string>
 <string>Field2</string>
 <string>Time1</string>
 <string>Low</string>
 <string>High</string>
 <string>FullName</string>
 </traits>
 <double>123456</double>
 <double>234567</double>
 <date>1274181389019</date>
 <double>100000</double>
 <double>200000</double>
 <string>Company Name Ltd</string>
 </object>
</object>

The binary AMF protocol strips out the XML tags which are replaced by single byte type markers (shown by the percent symbol in the message below). This cuts the message size a lot, but it is still pretty big.

AMF binary message = 403 bytes

1ae%%
 1a7%%
 %%%
 %flex.messaging.messages.AsyncMessage
 %destination
 %headers
 %correlationId
 %messageId
 %timestamp
 %clientId
 %timeToLive
 %body
 %market-data-feed
 %%%
 %DSSubtopic
 %PATH.TO.INSTRUMENT
 %
 %IC49204A4-D0D5-6C35-476C-B0170F6444AA
 %%%%%%%%%
 %IC47E1B72-01DF-9FE4-6EEE-D7D230F6A28E
 %%%%%%%%%
 %%%
 %flex.samples.marketdata.Stock
 %Field1
 %Field2
 %Time1
 %Low
 %High
 %FullName
 %123456
 %234567
 %%%%%%%%%
 %100000
 %200000
 %Company Name Ltd
 %%%%%

Conclusions

As you can see, message sizes can differ a lot, even when they are representing the same data payload. Message size may not seem that important at first, but if you want to deliver real time data to a lot of clients it quickly adds up to a large bandwidth bill and saturated networks which can affect hosting costs. Bandwidth on the client end may also be an issue and latency can be affected in some situations. Smaller is better!

6 thoughts on “Comet server message sizes, Bandwidth considerations”

    1. Well, I haven’t tried it out, but I can make a couple of assumptions with Kaazing.
      1. When it can actually use HTML5 WebSocket the message size will be down to the developer, since you will be creating the message yourself. WebSocket itself will add a couple of bytes only.
      2. When it is using its fallback mechanisms (ie with most browsers) it will probably have a bit more of its own envelope around the developers message – I’m guessing maybe at least a script tag wrapper – but the significant part of the message will be down to how the developer has chosen to represent the payload himself.

  1. Martin,
    Because you included Migratory in the list of Comet servers you published in your previous blog, I want to provide precise bandwidth information for Migratory Push Server as it is not present in this post.
    The most recent version of Migratory Push Server (version 3) is highly optimized for bandwidth usage and adds to each message the following overhead to actual data payload:
    38 bytes for Internet Explorer, and
    16 bytes for all the other browsers
    The overhead is constant as the protocol of version 3 does not use anymore field/value structures for messages but opaque data.
    Demonstrations for version 3 will be available on our web site in a few days. Also we will publish very soon the performances of Migratory Push Server 3, which will include charts for bandwidth usage. I will only mention here that with Migratory Push Server 3 we were able to push data to 1 million concurrent users from a single push server instance.
    Mihai

  2. Thanks Mihai, looking forward to seeing your latest performance results – we’re running some new tests ourselves at the moment.

  3. Martin
    Your analysis of Nirvana is incomplete and in some places incorrect.
    Protocols
    Nirvana supports a number of different protocols NSP, NHP, NSPS, NHPS etc. All are defined in this web page (http://www.my-channels.com/developers/nirvana/common/nativeprotocols.html) and all are capable of supporting the binary transfer of data to any one of our supported native client API’s. This table will provide your readers with API comparisons (http://www.my-channels.com/developers/nirvana/apicomparisons.html)
    Our Javascript library automatically detects an underlying browsers capability and uses either Comet or WebSocket to provide streaming services to a web browser. NHP/NHPS does not mandate the use of either Comet/WebSocket. They are highly optimized protocols for streaming real time content to clients via HTTP/HTTPS. This data will be binary unless the client is using a language that does not support binary (e.g. JavaScript).
    Verboseness of data
    Nirvana transparently marshals its events between all the languages and API’s that it supports. Some APIs ( like JMS ) require numerous headers to be included in each event payload and your example of a Nirvana payload is a good demonstration of this. All of these headers are configurable and controllable both programmatically using our administration API’s as well our graphical administration tool. The global configuration values found here (http://www.my-channels.com/developers/nirvana/enterprisemanager/realmadmin/realmcfg.html ) better describe the function of each header property.
    Event Deltas
    Nirvana supports sending event deltas through its merge engine. The feature is configurable and an example of such functionality can be found here (http://www.my-channels.com/developers/nirvana/enterprise/java/pubsub/eventdeltas.html). A user need not differentiate between an update and a full event. Nirvana will manage the deltas and send only what is required. This is very powerful mechanism for conserving bandwidth.
    Conclusion
    I agree with some of your points. However, if clients are really concerned with latency, bandwidth overheads and the efficient real-time delivery of content to RIA clients then their best bet is to opt for a platform with good binary protocol support. Nirvana provides this whilst also offering support for Comet/WebSocket based streaming for those who are prepared to send non-binary data over the wire.
    Finally, and as I mentioned in a response to your previous blog on Comet, Nirvana is so much more than Comet and this general categorisation you continue to make of RIA real-time messaging technologies as Comet is an inadequate way of describing products in this space. Of all the leading SDP platforms currently using Nirvana as their underlying communications platform none of them were prepared to incur the performance overheads associated with Comet and its lack of support for basic binary datatypes.
    Regards
    Paul

  4. Hi Paul,
    Thanks for the information.
    Are you saying that the various JMS and nvr fields can be configured out or programmed out? That would help a lot with the size of the Javascript messages.
    I’m not sure which parts of my post you are saying are incorrect? I have shown example messages based on using Javascript and Java – I was trying to be completely factual in this post, so I am happy to correct anything that is wrong.
    I’m aware that Nirvana is much more than a Comet server – so is Liberator, so are the others. This post is about Comet server functionality though – does every blog about any subject have to list everything else any software mentioned can also do?
    What do you see the performance overheads associated with Comet are?

Leave a Reply to Martin Tyler Cancel reply

Your e-mail address will not be published. Required fields are marked *