This article explains how Ably complements Kafka to reach end-users on the public internet.
Introducing Kafka
Kafka is a widely adopted open-source event streaming solution to capture, store and process event data. It uses the publish and subscribe (or pub/sub) pattern and acts as a broker to enable asynchronous event-driven communication between software components.
Kafka is popular for building high-performance, event-driven pipelines due to its scalability, delivery guarantee, node failure management, high availability, and fault tolerance. Let's now briefly summarize Kafka's key concepts.
Events
Events (also known as records or messages) are Kafka's fundamental building blocks. An event records that something relevant has happened, such as "Bob sent a message" or "Alice made a payment".
Topics
A topic is an ordered sequence of events stored durably for as long as needed. The various components of your backend ecosystem can write and read events to and from topics. Each topic consists of multiple partitions, which allows you to parallelize a topic by splitting its data across numerous Kafka brokers. Services within your system can simultaneously read and write data from and to multiple brokers for optimum scalability.
Producers and consumers
Producers are services that publish (write) to Kafka topics, while consumers subscribe to Kafka topics to consume (read) events. Since Kafka is a pub/sub solution, producers and consumers are entirely decoupled.
Kafka is designed to operate in private networks, enabling data streams to flow between microservices, databases, and other types of components within a backend ecosystem. It works best with a defined and predictable number of topics and consumers.
To enable Kafka to distribute messages to an unknown and rapidly changing number of clients on a public network like the internet, you will need an internet-facing messaging broker that sits between Kafka and your consumers.
| Read more | How to connect your Kafka pipeline to end-users at the network edge
Building and maintaining a proprietary solution to extend Kafka's functionality across network boundaries would be massive, complex, and costly. It's often more convenient and less risky to use an existing solution.
Introducing Ably
In a recent report, Gartner cites Ably as a leading example of a commercial event broker platform widely used for edge messaging. Edge messaging relates to streaming data across the public internet, and an edge messaging broker provides reliable event distribution between devices and datacenters — both from the device to the datacenter and from the datacenter to the device.
The Ably platform offers a scalable, dependable, and secure way to distribute and collect low-latency messages (events) to and from client devices via a fault-tolerant, autoscaling global edge network. With Ably, you can build dependable live and collaborative experiences at scale: like chat, asset tracking, live collaborative editing, alerts, notifications, or live score updates.
Kafka and Ably compared
Ably and Kafka are complementary services, but while Kafka is an internal event streaming platform, Ably is an edge messaging service. Ably not only matches the same guarantees, messaging semantics, and characteristics as Kafka but augments and extends these for event streaming over the public internet.
Kafka works best with a low number of topics and a limited and predictable number of producers and consumers. Think of it as a 'big fat pipe' designed to process high volumes of data.
Conversely, Ably can rapidly scale horizontally to an unlimited number of channels and millions of concurrent subscribers with no need to manually provision capacity. Think of it as millions of thin pipes.
Channels are at the core of Ably's functionality: publishers use channels to relay messages to any number of subscribers. Publishers and subscribers are entirely decoupled. Ably channels can be created dynamically on demand. When you integrate Ably with Kafka, the Kafka topics are directly mapped to one or more Ably channels. This allows you to accommodate any number of Kafka topics for any number of Ably subscribers.
Ably is equipped with Four Pillars of Dependability, to match Kafka's capabilities:
Performance: Ably offers low latencies of less than 65 milliseconds round trip for the 99th percentile, which translates to high-grade performance.
Integrity: Ably guarantees message ordering and exactly-once semantics from publishers to subscribers.
Reliability: Ably ensures that data streaming through it is unaffected even during multiple outages.
Availability: The real-time messaging platform has a 50 percent capacity margin for instant volume spikes and offers a 99.999 percent uptime SLA.
What about security?
Kafka's security capabilities are designed for private networks. On the other hand, Ably is intended for secure event streaming across network boundaries, augmenting Kafka's security features. Ably offers multiple security mechanisms, including:
- SSL/TLS
- 256-bit AES encryption
- Token-based and basic key-based authentication
- Privilege-based access
- DoS protection and rate-limiting
Ably is also HIPAA compliant. Experity, a leading provider of integrated technology solutions for urgent care providers, chose Ably for its ability to work seamlessly with Kakfa to deliver critical data to realtime dashboards.
The Ably Kafka Connector
Ably's Kafka Connector is a sink connector that maps Kafka topics to Ably channels to move data easily into Ably.
Kafka Connector enables hassle-free integration between the two pub/sub tools and allows you to publish Kafka's real-time data to an extensive subscriber list while ensuring data protection.
Where next?
Find out more about how Ably helps you effortlessly and reliably extend your Kafka pipeline to end-users at the edge, or get in touch and let’s chat about how we can help you maximize the value of your Kafka pipeline.
Further reading
- Confluent Blog: Building a Dependable Real-Time Betting App with Confluent Cloud and Ably
- Extend Kafka to end-users at the edge with Ably
- How to stream Kafka messages to Internet-facing clients over WebSockets
- Dependable realtime banking with Kafka and Ably
- Building a realtime ticket booking solution with Kafka, FastAPI, and Ably
