We are excited to announce our new Kafka connector. The Ably Kafka Connector provides a ready-made integration between Kafka and Ably, helping companies distribute data from Kafka to internet-connected client devices in a fast, easy, dependable and secure way. As part of our partnership with Confluent, the connector is available on the Confluent Hub as a verified Gold connector.
By 2025, it is projected that 30% of all data will be realtime. Data created inside companies is on the rise, and will amount to half of the realtime datasphere, with the other half being generated by consumers. In the next 5 years, relevance - both for market incumbents and challengers - will depend on how well realtime data is exploited to provide the personalized, instantaneous, on-the-go experiences all customers have come to expect.
When it comes to streaming and processing huge volumes of mission-critical, time-sensitive event data, Kafka is the de facto choice for most companies. But Kafka is not designed for distributing events between internal systems and consumers on the public internet. Engineering teams building products that stream realtime event data to millions of user devices face the challenge of finding solutions for extending Kafka to the edge.
The solution requires using an Internet-facing messaging layer to sit between Kafka and client devices. Building one is technically challenging and hugely time consuming, as it requires significant DevOps and engineering resources spent on building and maintaining public-facing interfaces and provisioning application servers for load balancing and high availability. However, there are ways to solve the problem of message distribution from Kafka to client devices, and developers are increasingly turning to them. Ably is one such way.
Extending Kafka without the pain
The new Ably Kafka Connector is a sync connector built on Apache Kafka Connect, which can be self-hosted or hosted with a third party provider - the most common being the Confluent Platform. The connector, now released in beta, is verified by Confluent as Gold, following the guidelines set forth by Confluent’s Verified Integrations Program.
Ably is designed for streaming messages at extreme scale to web, mobile, and IoT devices, via a fault-tolerant, dynamically elastic global edge network. Many of our customers use Ably as an Internet-facing extension of Kafka to send mission-critical event data to consumers across network boundaries in realtime.
Using the Ably Kafka Connector, you can achieve seamless integration between Ably and your existing setup to easily publish Kafka realtime event data:
- To a large, rapidly changing number of end-user devices over the Internet
- Without risking the security of your company’s data
- At dependably high standards similar to those provided by Kafka
Ably is naturally complementary to Kafka, and offers the same guarantees, messaging semantics, and characteristics:
Among the most common use cases we see for extending Kafka to the edge are:
- Financial and banking cloud-based mobile products sending time-sensitive information such as stock data or currency exchange rates to customers.
- Medical care and emergency services where workers must receive reliable, instant information of emergencies across multiple locations, so urgent task forces can be deployed where they are most needed.
- Audience engagement apps and platforms fanning out realtime scores, updates and commentary to millions of devices.
Scale data streaming to millions of end-users without infrastructure provisioning
Kafka works best with a low number of topics, and a limited and predictable number of producers and consumers. Additionally, it is not designed for fine-grained data distribution and doesn’t have a mechanism to ensure that when a client device connects over the Internet, it only receives the messages that are relevant for that user/device.
Ably, on the other hand, delivers over 550 billion messages in sub 65ms to more than 250 million end-users each month via 15 datacenters and 205 edge acceleration points-of-presence. Our platform can rapidly scale horizontally to an unlimited number of channels and millions of concurrent subscribers - with no need to manually provision capacity and a 50% capacity margin for instant surges.
Ably’s channels (the equivalent of Kafka topics) are optimized for cross-network communication and allow for flexible routing of messages from Kafka topics, ensuring that clients connecting over the Internet only subscribe to relevant information.
Create dependable digital experiences for users at the edge
Inside your network, Kafka guarantees delivery and message ordering, high availability, no single point of failure, durability and low latency. But, in order to provide these guarantees, Kafka requires reliable, constant network conditions. This is simply not possible when sending data over the public Internet, where the quality of data networks is unreliable.
Ably is designed to provide critical functionality at scale across networks of varying reliability and can guarantee:
- Message ordering, guaranteed delivery, and exactly-once semantics, from the moment a message is published to Ably, all the way to its delivery to consumers.
- Predictable 65ms round trip latency for the 99th percentile.
- Fault tolerance at regional and global levels for service continuity even in the event of multiple infrastructure failures, with 99.999999% message survivability.
- A 99.999% uptime SLA.
Keep your company network completely secure
Allowing your company’s data-processing component to be directly accessible over the Internet can pose serious security risks. Finding a way to expose your Kafka data at the edge in a secure way is part of the challenges engineering teams face when building solutions for extending Kafka over the public internet.
Using Ably as a Kafka extension to the edge completely decouples the backend Kafka deployment from end users. Clients never connect directly to Kafka. Instead, they are only allowed to subscribe to the Ably channels that are relevant to them.
Unlike Kafka, Ably was designed with multiple security mechanisms for data distribution across network boundaries, from network-level attack mitigation to individual message-level encryption:
- SSL/TLS Encryption
- 256-bit AES Encryption
- Token-based and basic key-based authentication
- Privilege-based access
- DoS protection & rate limiting
- HIPAA and EU GDPR compliance
- SOC2 Type 2 certified
See the Ably Kafka Connector in action
The best way to understand how something works is to see it in action, so Ben Gamble used the connector in a ticket booking solution that processes and distributes large quantities of ticket data to and from a large number of customers. To build this solution, he used FastAPI, Kafka and Ably and a few other technologies - all freely accessible. To see how it works and to follow along if you want to, see this step-by-step guide.
Technical guide: Build a realtime ticket booking solution with Kafka, FastAPI, and Ably
Getting started with the Ably Kafka Connector
The Ably Kafka Connector works with any type of Kafka deployment that includes Kafka Connect, be it self-hosted, fully managed, on-prem, or in the cloud. At the moment, the Ably Kafla Connector allows you to:
- Publish Kafka records from one or more Kafka topics into a single Ably Channel
- Connect to Ably using an API Key
The Ably Kafka Connector is a verified Gold connector on the Confluent Platform. It can also be deployed locally using Docker. For more information on deployment and configuration, check out the documentation.
The connector is available under the Apache 2 open source license and we are planning to continue extending and improving it, so we encourage feedback and feature requests; please either raise issues or pull requests if you would like to talk about contributing or feature requests. You can also contact us at any time.
If you’re looking to learn more about the various methods and features available on the Ably platform, you can create a free Ably account to get started and take a look at our general documentation. For any questions or feedback, please get in touch.
Ably and Kafka resources
- Confluent Blog: Building a Dependable Real-Time Betting App with Confluent Cloud and Ably
- Extend Kafka to end-users at the edge with Ably
- How to stream Kafka messages to Internet-facing clients over WebSockets
- Dependable realtime banking with Kafka and Ably
- Building a realtime ticket booking solution with Kafka, FastAPI, and Ably
- Ably Kafka Connector
- Ably’s Kafka Rule
- Introducing the Fully Featured Scalable Chat App, by Ably’s DevRel Team
- Building a realtime chat app with Next.js and Vercel
- Ably pub/sub messaging
- Ably’s Four Pillars of Dependability
