I am a self-professed geek and get a buzz from solving difficult technical problems. Given the option, I’d be solving the world’s problems one tap a time on my keyboard ;) However, as the co-founder of Ably, a realtime data delivery platform, a lot of the time I am talking to our customers and helping them solve their realtime data delivery challenges.
Ably has had a lot of traction recently with fantasy app developers, and I’m starting to see common challenges for these businesses. As a firm believer and contributor to open source, in this post I explore some common problems we see for fantasy sports and live events app builders, and share some design patterns that I hope will be useful for other developers.
Problem 1: Unknown and theoretically limitless number of players receiving live updates
Most often a stream of events arrive at the app developer’s servers from a sports data provider (such as Opta or Sportradar), some business logic is applied, and the data then needs to be pushed to every device participating in that game.
For example, when a goal is scored, you may want every user of your app to receive that update simultaneously within 200 milliseconds anywhere on earth. The challenge typically is that you want that goal to be met when you have one or a million users, without any change to your architectural design.
Pattern 1# — Pub/Sub
This pattern is hardly new, it’s been around since 1987, yet it’s still a good way to approach asynchronous realtime data distribution. The pattern involves two roles, Pub represents the publisher, Sub represents the subscriber. The pattern specifies that the publisher (your server) publishes messages without any knowledge of how many subscribers, if any, there may be. And subscribers register to receive message without any knowledge of of the publishers. This ensures the two roles are intentionally decoupled with a middleware broker responsible for receiving messages and fanning out the messages to the subscribers.
How does this help?
It ensures your apps scale to theoretically limitless subscribers without any changes to the design of your system, thus keeping your stack simple whilst giving you the scale you need. The middleware broker is responsible for providing scale. If you chose a middleware solution that has proven scale, then by adopting the pub/sub pattern, as you scale only one component needs to address that need in a predictable way.
Our customers use Ably’s Pub/Sub feature as the middleware broker which provides limitless scale. Channels are used to provide topic filtering such as one channel for each game or player. As Ably’s system is inherently elastic, developers trust us to look after the scale issues.
Problem 2: Data synchronisation
Problem 1 describes how data, as messages, is distributed to devices, but it does not address how you keep the game data in sync consistently with all devices.
For example, your app may need to maintain live stats for each player. As every event occurs during the game, your app needs to reflect that change in real time both in the UI and also within the local storage. The challenge is one of data integrity and bandwidth. If you publish the entire game data every time an event occurs, it is inefficient and will result in huge bandwidth bills as you scale. Perhaps more importantly this could impair the user experience for people on slower connections or with expensive bandwidth. If however you only send data updates, how do you ensure that the data integrity is maintained i.e. you need all updates arrive reliably and in order?
Pattern 2.1# —Serial JSON Patches
JSON Patch is a standard that defines how you can send only the deltas for a JSON object as it mutates. For example, if you had a table of all players with their stats, and only one player’s stats changed following a goal being scored, then the patch may look something like:
[ { "op": "replace", "path": "/player/bob/goals", "value": "1" },]How does this help?
JSON Patch provides a means to efficiently send deltas for a JSON object thus reducing bandwidth overhead significantly. However, JSON Patch does not provide the complete solution as:
- You need to obtain the JSON object at the outset
- The JSON Patches must be applied in the exactly the order they were generated — a missing or out-of-order patch will result in complete loss of data integrity
Ably, uniquely in the realtime messaging industry, offers reliable data delivery uniquely ensuring that data arrives in the correct order and continuity is assured. We also provide a message history (replay) feature providing a means to obtain historical message published on the channel prior to connecting. Finally, we uniquely offer continuous history ensuring developers can reliably obtain history and receive subsequent realtime updates without any missing messages or duplicates.
A pattern we’ve seen developers use with Ably to solve this problem therefore is:
- Configure a channel to persist messages
- Publish the original JSON object on the channel, and then subsequently all JSON Patches
- Clients when connecting then obtain the channel history and subscribe to future JSON patches. The history provides a means to build the JSON object from the initial object plus all the patches, and the attached channel ensures live updates continue to be received in order with integrity.
- If a client loses continuity on the channel (this may happen if the client is disconnected for more than two minutes), the app simply repeats the previous step.
Ably will be releasing an efficient way to stream updates to data objects natively in our client libraries in the future. If you are interested in using this feature, please get in touch.
Pattern 2.2# — CRDT
A CRDT is a conflict-free replicated data type. Unlike JSON Patch, it allows multiple parties to concurrently updates the underlying data object. Each update is then distributed to all other parties, and the algorithm ensures that once all updates are applied by all participating parties, the underlying data object will become eventually consistent, regardless of the order the updates are applied.
CRDTs provide a very sophisticated way to ensure data is consistent, even when there are multiple parties changing the data at the same time. However, in order to provide the eventual consistency guarantees, there are limited data types with specific restrictions.
We find CRDTs are more commonly used in collaborative applications similar to Google Docs, where multiple users can update the content simultaneously. Riak is one of the few database solutions that provides native CRDT support. I will leave it to the reader to consider whether CRDT is appropriate for their use case and how best to implement it.
Problem 3: Upstream game play events
Problem 1 and 2 addresses the issues of scaling downstream data to your devices. However, participating in fantasy games will have an upstream component i.e. users on devices may change their team configuration, place a bet, or simply want to chat with other players.
Typically this is handled with a simple HTTP request to your servers which in turn run some business logic which may respond synchronously (as part of the response to the HTTP request), or asynchronously (pushed back to the device later as a message).
The problem app developers face are:
- Using an HTTP request in a synchronous fashion increases the likelihood that the operation fails in changing network conditions. For as along as the client is waiting for a response, there is a chance the connection state will change and the underlying TCP connection for the HTTP request will be closed. If a request has failed due to a TCP connection, it may need to be retried, but unless the operation is idempotent, it could result in unexpected behaviour for users.
- If there is a sudden spike of activity, perhaps due to an unforeseen change in the game such as a player being taken off, then your servers will need to absorb that load immediately. You need to predict the load in advance of each game and ensure you have sufficient service capacity for the spikes.
- HTTP provides no ordering guarantees i.e. a later request may arrive before an earlier request when they are close together. Sometimes this matters.
Pattern 3# — Message queues
Traditional message queues are designed to provide a message-orientated means to communicate between components of the system, specifically the devices and your servers in this instance. Like the pub/sub middleware, the message queue acts as a broker distributing messages to each consumer. If your messaging middleware is scaleable and resilient, then decoupling each component in the system ensures failure or overload in one area of the system will not affect any other part of the system.
Please note, unlike the pub/sub pattern where each message is delivered to all subscribers, message queues typically deliver messages only once to one of the consumers (technically AMQP can only provide an :)at least once guarantee, however in almost all situations it provides exactly once delivery). Messages typically operate with FIFO (first-in-first-out) ensuring that in spite of any backlog that builds up if consumers cannot keep up, the consumers will always be working on the most oldest messages first.
How does this help?
By decoupling publishers of data (your devices in this instance) from the consumers of the data (your servers), and introducing a message queue middleware broker, you are building a fault tolerant system. For example:
- Huge spikes of activity may slow down the responsiveness for customers as the workers have to work through the backlog of messages to process, but customers will not experience failures.
- Where the ordering of events from device to server is important, the message queue is able to provide ordering guarantees. The Ably service too provides reliable ordering, so ordering is maintained from publishing to Ably all the way through to the queue.
- Messages from devices are streamed immediately into the queue and the device receives an ACK immediately notifying the device that the message has been received. As a result, if you use Ably for publishing, we can ensure a message is never published twice (i.e. if you publish a message, lose connection before you receive an ACK, and then retry the message once connected, we de-dupe that message).
Ably provides message queues as a service as part of our realtime data delivery platform. Get in touch if you would like access to our queues.
In this article I have touched on three common realtime challenges that we see for fantasy sports app developers. There are many more and I’d be happy to address these in a future blog post.
Please do get in touch if you’d like to discuss this article, your realtime challenges or have any other questions.
