Introducing Message Delta Compression: Bandwidth-efficient realtime messaging

Ably’s philosophy is to abstract away the development and operational complexity of building realtime capabilities. We strive to provide a platform that developers can depend upon to handle even the tiniest of realtime details, so you can focus on what really drives your product forward. Let me introduce Message Delta Compression.

Message Delta Compression reduces the bandwidth required to transmit realtime messages by sending only the changes from the previous message to subscribers each time there’s an update, instead of the entire message. Let’s say the first message sent over a channel is 16KiB and the message after that is the same but with an additional 4KiB of new data. With Delta Compression that message will have a payload of only 4KiB instead of 20KiB. This reduces bandwidth costs, improves latencies for end-users, and simplifies development by taking care of message size optimization.

In order to use Message Delta Compression clients must specify parameters on a per channel basis through a new ChannelOptions API, available today as part of our Client Library SDKs v1.2 release. The libraries available in this 1.2 release are: JavaScript, Java, Cocoa, and .NET. Check our Client Library SDK feature support matrix for full coverage.

Message Delta Compression is a stable feature: we expect it to be forwards-compatible with future Ably updates. We encourage you to read the entirety of this announcement but if you want to jump right in, check out the documentation for Message Delta Compression and ChannelOptions.

The problem Message Delta Compression solves

A major use case for realtime messaging is synchronizing state changes with subscribers. The easiest way to do this is resending the entire state every time there’s a change. Yet constantly resending the same data, when only a small portion has actually changed, bloats a message with older updates. This increases the bandwidth required to transmit it, raising developer costs and degrading end-user experiences with increased latencies.

This exposes developers to the complexities of transmitting realtime messages with large payloads, forcing them to think about how they manage message size when instead the platform should be abstracting these complexities away.

Message Delta Compression reduces the bandwidth required to transmit messages by sending only the changes from the previous message to subscribers each time there’s an update, instead of the entire message. Developers can send the entire message payload and Ably’s diff algorithm takes care of generating a delta that maintains message and stream ordering.

Ably, like all other realtime messaging platforms, has limits on the maximum message size. For free accounts Ably's limit is 16KiB and for paid accounts 64KiB - double that of other platforms. These limits still apply but Message Delta Compression makes it easier to send the entire message payload for longer.

Let’s take a tennis match where the organizer needs to send an update each time there’s a new point scored. The first message payload might be 16KiB. But as the match goes on and more points are scored the size of that message grows as it contains all previous updates. By the middle of the tennis match the message payload could be 64KiB. But each individual update might only be as small as 4KiB.

With Message Delta Compression, if the entire message payload is 64KiB but the content that differs from the previous message only equals 4KiB, then Ably generates and sends a delta message based on just the content that differs. In this case, the delta message sent would be only 4KiB instead of a 64KiB non-delta message.

This reduces your bandwidth costs and improves latencies for end-users, providing a better, more performant experience for them. And it simplifies development with Ably as you’ll rarely need to think about the implications of large messages, such as splitting different state objects across multiple channels.

Message Delta Compression using an Ably Client Library SDK

As a publisher using an Ably Client Library SDK you still publish the entire message over Ably. Ably’s diff algorithm encodes the difference between the new payload and previous payload, and outputs the set of differences between them. That output is called a delta and is what Ably sends to subscribers. Note that deltas are only applied to the principal payload, not the metadata (e.g. time stamp).

Ably generates deltas using the open VCDIFF format. We then use an opaque binary algorithm to decode the delta: the open source xDelta algorithm. It’s not within the scope of this announcement to dive further into the detail of diff algorithms or VCDIFF. But if you want to learn more, check out our Practical Guide to Diff Algorithms and Patch File Formats.

If you’re using Message Delta Compression and want to know what your bandwidth savings are, you can find that information in your Ably dashboard. Right now we show only what the total non-delta bandwidth usage would have been without Delta Compression.

How to use Message Delta Compression with ChannelOptions

When we released Channel Rewind we introduced channel parameters, expressed as a query string in the connection request or channel name itself. This was a first step to specifying parameters on a per channel basis. With Ably’s 1.2 Client Library SDKs, Ably's ChannelOptions interface has been expanded to include not just end-to-end encryption parameters but also channel parameters. There are now three parameters:

• delta to enable Message Delta Compression
• rewind to enable and specify Ably’s existing Rewind functionality
• cipherParams to manage encryption on a channel

Subscribers can now opt to subscribe in Delta Compression Mode using the new ChannelOptions interface. This is handled on a channel-by-channel basis.

Here's an example in Java:

AblyRealtime ably = new AblyRealtime("xVLyHw.cL-rhQ:IMFqIOPDcEyrpGty")
Channel channel = ably.channels.get("wax-dip-dam", new ChannelOptions{{params = Map.of("delta", "vcdiff")}});
channel.subscribe(new MessageListener() {
  @Override
  public void onMessage(Message message) {
    System.out.println("Received `" + message.name + "` message with data: " + message.data);
  }
});

Considerations for subscribers

Ably constructs a delta based on the similarities between consecutive messages, generating and sending a diff based on changes since the previous message. But there are scenarios where a subscriber always get a full message:

• When a client first subscribes to a channel, the first message they receive is always a full message. Any subsequent messages are then sent as deltas.
• Ably won’t send a delta when it comes to stream discontinuity scenarios. If clients are disconnected or dropped from Ably for any reason, the first message upon reconnecting will always be a full message.
• Ably will send a full message if generating a delta doesn’t result in significant (>20%) bandwidth reduction.

A note on ably-js and Message Delta Compression

We're aware of Ably’s JavaScript Client Library SDK footprint and we're actively avoiding bloat. As such we’ve split out the VCDIFF decoder from the core of ably-js. In order to subscribe in Delta Compression Mode using ably-js you need to plug in a separate VCDIFF decoder library. This is hosted on Ably’s CDN. We will be modularizing this in the future, but right now this is something you will need to consider.

/* Make sure to include <script src="//cdn.ably.io/lib/vcdiff-decoder.min-1.js"></script> in your head */

var realtime = new Ably.Realtime({key: 'xVLyHw.cL-rhQ:IMFqIOPDcEyrpGty', plugins: {vcdiffDecoder: vcdiffDecoder}});
realtime.channels.get('wax-dip-dam', {
  delta: 'vcdiff'
}).subscribe(msg => console.log("Received message: ", msg));

Message Delta Compression using a non-Ably library

If you need to use a non-Ably library, say to stream over MQTT or Server-Sent Events, and want to use Delta Compression you’ll need to reconstruct the messages yourself using the deltas. It’s a little trickier than using an Ably Client Library SDK.

Delta Compression relies on encoding and decoding to generate bandwidth-efficient deltas. When using a non-Ably library you must decode messages yourself - encoding is still handled for you. Ably provides a VCDIFF decoder library to do so for Java, Cocoa, .NET, and JavaScript. The limitations with this is that you must use channel qualifiers.

Here's an example using Server-Sent Events:

var key = 'xVLyHw.cL-rhQ:IMFqIOPDcEyrpGty';
var channel = 'sample-app-sse';
var baseUrl = 'https://realtime.ably.io/event-stream';
var urlParams = `?channels=${channel}&v=1.1&key=${key}&delta=vcdiff`;
var url = baseUrl + urlParams;
var eventSource = new EventSource(url);
var channelDecoder = new DeltaCodec.CheckedVcdiffDecoder();

eventSource.onmessage = function(event) {
    /* event.data is JSON-encoded Ably Message
       (see https://www.ably.io/documentation/realtime/types#message) */
    var message = JSON.parse(event.data);
    var { id, extras } = message;
    var { data } = message;

    try {
        if (extras && extras.delta) {
            data = channelDecoder.applyBase64Delta(data, id, extras.delta.from).asUtf8String();
        } else {
            channelDecoder.setBase(data, id);
        }
    } catch(e) {
        /* Delta decoder error */
        console.log(e);
    }

    /* Process decoded data */
    console.log(data);
};

State Persistence is now available

A common request was the ability to always-persist the last state on a channel, a little like indefinite history.

You ask, we listen, we act: you can now persist the last realtime message on a channel for 365 days. This is beneficial for channels that have either a variance in frequency or low frequency of messages. For example, a typical use case would be a dashboard showing a market or currency rate data - the software will have the last data to show immediately after subscribing to a channel, before any new updates occur.

To enable state persistence on a channel, log into your Ably account and enable Persist last message on Channel rules to store the last published message for 365 days. Retrieving the last message can be achieved using rewind=1 on the rewind API.

State persistence arrived in February 2021.

Get started with Message Delta Compression

To get started visit the Message Delta Compression documentation. As always, please get in touch if you have any questions.