Channel Rewind and State Persistence

One of the driving forces behind Ably is to make the complex simple. Take our APIs, which package Ably’s realtime messaging fabric and global cloud network into standardized programmatic interfaces, making it easy for developers to build demanding realtime apps or distribute data streams to other developers. But we’re always working to improve.

Let me introduce Channel Rewind, an optimization for Ably's Pub/Sub Channels. Rewind simplifies the process of retrieving realtime messages sent over a channel prior to connection. It effectively enables you to connect to a channel, instantly load the state, and subscribe to all future realtime updates with a single API interaction.

If you want to jump right in, check out the:

• New Rewind sections in Channel Parameters, Channels, and History documentation
• JavaScript tutorial using Ably’s JavaScript SDK
• Server-Sent Events (SSE) tutorial
• MQTT tutorial

What is Rewind?

As a pub/sub messaging platform, our APIs enable clients to subscribe to a channel and receive realtime messages from that point onwards. But it's sometimes useful to access messages from the past.

Let’s take the example of a channel delivering weather updates. Updates might only be published once every 15 seconds. If a client connects to a channel just after a publish, you don’t want them to wait 15 seconds for the next update - you probably want them to see the most recent update so they’re not left hanging.

Ably has always supported retrieval of previously-sent messages with our History API. In order to get any messages sent on a channel before connection you would need to make an additional History API query once connected. However, this adds additional latency along with complexity, especially for clients not using Ably Client Library SDKs, such as our MQTT or SSE protocol adapters.

The process of retrieving previously-sent messages until now:

1. Client makes a channel attach request
2. Attach request succeeds, messages start arriving in real time
3. Client makes a history API request for messages sent from a specified point in the past up to the point of the attachment
4. History request succeeds, and the client has to integrate the results of the history request with the live message stream

With the Rewind implementation, you can now pass a parameter along with a connection request to specify that you want to see previous messages upon connection. You can effectively load the current state of a channel and request all future updates in a single request. This means one less API call, lower latency and bandwidth usage, and a more seamless experience.

The process after Rewind:

1. Client makes a channel attach request that includes a rewind parameter, specifying a point in the past to retrieve messages from
2. Attach request succeeds, and the backlog of messages arrives immediately afterwards, after which new messages start arriving in real time

The backlog seamlessly blends in to the live message stream, with no message loss or duplication and all messages arriving in the same order they were sent, just as if you had actually been attached since the requested rewind point.

How does Rewind work?

As above, you can now pass a parameter along with a connection request to specify that you want to see new messages upon connection. There are currently two types of parameters:

• The number of messages to retrieve. For example rewind=10 would provide the last ten messages published on a channel. The limit is 100. If the request goes back far enough it can reach into persisted history (provided you have it enabled).

var realtime = new Ably.Realtime('xVLyHw.qahRdQ:6pCpxE153FuyKd0y');
var channel = realtime.channels.get('[?rewind=10]pan-hug-fur');
channel.subscribe(function(message) {
  alert('Received: ' + message.data);
});

• The amount of time in the past. For example rewind=10s would provide all messages sent within the last ten seconds. And rewind=2m would provide all messages sent within the last two minutes. The max can rewind is up to two minutes. For now, it's not possible to reach into persisted history using time-based parameters. If you'd like the ability to do this, please get in touch.

var realtime = new Ably.Realtime('xVLyHw.qahRdQ:6pCpxE153FuyKd0y');
var channel = realtime.channels.get('[?rewind=10s]pan-hug-fur');
channel.subscribe(function(message) {
  alert('Received: ' + message.data);
});

There are three ways to use Channel Rewind:

• Use the channel parameters option in the Ably Client Library SDKs.
• Add channel parameters in a channel name itself. This is especially useful for SSE or MQTT as they don’t use an Ably SDK.
• Query string parameter for HTTP based connections. If using this method it applies to all channels a client is subscribed to. For example, you can add rewind=1 parameter to an SSE stream and all channels on that stream would rewind the first message upon connection.

Read the docs for more explicit detail on using Rewind.

Considerations and limitations:

• Rewind is currently limited to 100 messages so your connections don’t get overwhelmed. If you want to retrieve more than 100 messages (or if you want pagination) you will still need to carry out a history request.
• It's only possible to rewind by up two 2 minutes.
• If using Rewind you will receive all ‘rewound’ messages before realtime messages. This is the same as Ably’s stream resume feature.

Some example use cases for Rewind

As Rewind allows you to instantly load current state and request all future state changes in a single request, the uses are varied. Here are some examples of when you might implement Rewind.

A channel that holds location state

A channel providing weather updates may only receive new data once every 15 seconds. A client that connects one second after the most recent update would need to wait for the next update - or carry out a History API request - before receiving any weather information. But with Rewind they can instantly receive the most recent state of that channel - in this context, the most recent weather update.

Transport also works. For example, a channel that provides the platform number for the next train arrival. Clients receive the latest platform number while also subscribing to any new updates.

A chat application

You might have public chat in a multiplayer game. It makes sense to provide a connecting client with 30 seconds of previous messages so they have some context on what’s been happening in-game before they joined.

Get the latest state of an IoT device

Let’s say it’s Winter. You’re about to leave work and want to come home to a warm house. You check the temperature of your house using an app connected to a thermostat. In this scenario, you’d want to know the current temperature in your house (or the state of the temperature channel) instantly upon connection.

Rewind + Message Delta Compression

Rewind was originally part of the work we were doing to implement Message Delta Compression, which reduces the bandwidth required to transmit realtime messages by sending only the changes from the previous message to subscribers each time there’s an update, instead of the entire  message.

Let’s say the first message sent over a channel is 16KiB and  the message after that is the same but with an additional 4KiB of new  data. With Delta Compression that message will have a payload of only  4KiB instead of 20KiB. This reduces bandwidth costs, improves latencies  for end-users, and simplifies development by taking care of message size  optimization.

Message Delta Compression arrived June 2020.

Rewind + State Persistence

State persistence goes beyond getting the last state of an IoT. The limitations of the rewind and history API is the retention period. Instead of sending the same state frequently to a channel from a device, it is optimal to send changes in state to reduce the number of messages. If the state hasn’t changed in a week, the last published message would be lost.

Now with state persistence, you can store the last published message for 365 days for specific channels. For example, in instant messaging apps you have information for the user, such as status or last seen. If a user goes offline for a few weeks, you can still access their last state upon connection.

State persistence arrived in February 2021.

Get started with Rewind

To start using Rewind, visit the documentation or tutorials. As always, please get in touch if you have any questions.