Talos Basic Concepts

Basic Concepts

The Talos architecture is as shown in the diagram. Producer continuously produces messages and sends them to the assigned Topic; Consumer reads the messages inside the Topic and consumes them. The Topic messages are distributed to different partitions.

Topic & Partition

Topic is a logical Elastic Message Queue, consisting of one or more partitions. The number of partitions determines Topic's maximum concurrency.

The number of partitions must be assigned when the Topic is created. After the Topic is created, the number of partitions can only be increased, they can't be decreased.

NOTE: When assigning the number of partitions, please consult the standards listed below for configurations. Each partition permits a maximum write of 512 K/s and a maximum read of 1M/s. Please set up a number of partitions appropriate for your amount of business data.

TopicName & TopicTalosResourceName

• TopicName is the Topic's name, and is assigned when the Topic is created. There can only be one Topic with the same TopicName on the system at any time.

• TopicTalosResourceName is a Topic's unique identifier after being created. Please note, if a Topic is created and then deleted, and a new Topic is created using the same name, their TopicTalosResourceName will not be the same.

• After a Topic is created, all subsequent read/write operations will use the TopicTalosResourceName as a unique identifier.

• The format for TopicTalosResourceName is: developerId#topicName#UUID

TalosAdmin

TalosAdmin is Talos's data control interface. It is used for DDL operations such as creating and deleting Topics. For detailed information, consult TalosAdmin API.

SimpleProducer

SimpleProducer is a synchronous interface for transmitting data. Each SimpleProducer corresponds to only one partition. For detailed information, consult SimpleProducer API.

SimpleConsumer

SimpleConsumer is a synchronous interface for consuming data. Each SimpleConsumer corresponds to only one partition. For detailed information, consult SimpleConsumer API.

NOTE: SimpleProducer and SimpleConsumer are low-level API provided by Talos for reading and writing data; they both use synchronous mode to transmit/read back data. In contrast, Talos also provides a high-level API, TalosProducer and TalosConsumer; these both use asynchronous mode to transmit/read back data.

TalosProducer

TalosProducer is a high-level API for transmitting data. Based on the Topic's number of partitions, it creates a corresponding number of threads to transmit data, and each thread has a queue that it utilizes to cache the user's data. TalosProducer uses asynchronous mode, and supports the user registering callback to handle the results of data transmission.

Key Points

• PartitionKey

If the user assigns a PartitionKey within a Message, TalosProducer will map the PartitionKey to a 32-bit positive integer, then distribute a corresponding number of partitions based on that integer.

If the user does not assign a PartitionKey within a Message, TalosProducer will randomly assign the Message a partitionID. The strategy for assigning partitionIDs is configurable; for specifics you can consult TalosProducer API and Configuration optional configurations scenario 4.

• Buffer

As mentioned above, TalosProducer is an asynchronous device, it can take user messages and cache them inside a queue. It should be noted that, if the client process hangs, data within the buffer may be lost. Users of the high-level API, please be informed of this.

TalosConsumer

TalosConsumer is a high-level API for consuming data. It can automatically run Re-balancer, automatically Commit messages, users can also adjust the configurations to personally control the messages' Commit logic. There are a few key concepts regarding TalosConsumer that must be understood, the specifics are as follows:

• TalosConsumer & ConsumerGroup

Each TalosConsumer is part of a special consumer group. When you initialize, you can assign the ConsumerGroup name. Please note, at any time, a given partition within a Topic can only be consumed by one consumer from the same consumer group, but multiple consumer groups can consume that partition at the same time.

As shown in the diagram, Partition 1 is currently being consumed by Consumer-1 from Consumer Group A, therefore Consumer-2 from Consumer Group A cannot consume Partition 1. However, as shown on the right side of the diagram, Partition 1 can be consumed by Consumer-1 from Consumer Group A and Consumer-1 from Consumer Group B at the same time.

• Re-Balancer

Re-Balance means that TalosConsumer can automatically assign consumption of a Topic's partitions based on the number of instances the user has started. If the number of TalosConsumer instances changes, and a TalosConsumer goes online or offline, TalosConsumer will automatically perceive this, and recompute the assignment of partitions.

As shown in the diagram, at the outset, there is only one TalosConsumer-1 process, and the Topic's five partitions are all being consumed by TalosConsumer-1. When the user starts another, similar consumer group's consumer instance, after TalosConsumer-2 comes online, TalosConsumer-1 will automatically perceive that TalosConsumer-2 is online and perform a calculation to free up two partitions. Then, TalosConsumer-2 can take over consumption of the two partitions that were freed by TalosConsumer-1.

For a detailed introduction to TalosConsumer and how to personally control the messages' Commit logic, please refer to TalosConsumer API.