TalosProducer API & Conf

TalosProducer API

addUserMessage(List<Message> messageList)

Parameter: messageList, the function accepts a Message List. The user can set PartitionKey(optional) for each Message to indicate which partition to send the message to. If not specified, the system will randomly specify the partition

Return value: Empty, the result of the message sending is the user-registered Callback

Exception: ProducerNotActiveException. This exception indicates that the producer has not initialized yet and a lower probability occurs. If it occurs, try again

TalosProducer configuration description

The following is the configuration description of TalosProducer:

Required configuration items

Optional configuration item & scenario (for High Level TalosProducer)

[Scenario 1] Client memory is not very large. When using Producer, it is not possible to cache too much data. He hopes that the cached data will be less, or he hopes to increase the cached data;

• galaxy.talos.producer.max.buffered.message.number

  Total number of buffers that can be buffered in TalosProducer memory. If the total number exceeds this configuration number, it will temporarily block interface addUserMessage until the remaining number is less than this value

  Default: 1000000

• galaxy.talos.producer.max.buffered.message.bytes

  Same as galaxy.talos.producer.max.buffered.message.number, which limits the total buffer's message bytes from the total number of bytes

  Default: 500MB

[Scenario 2] Producer will cache data, accumulate batch to send, and want to change some mechanisms of the buffer (for example, he wants to send faster / wants to accumulate a larger batch); any one of the following conditions is satisfied, and the batch will be sent

• galaxy.talos.producer.max.buffered.milli.secs

  This configuration limits the amount of time that a message is cached in client memory before it is sent to a partition; if this time is exceeded, it will be sent out. On the one hand, one can accumulate as many messages from a batch as possible. On the other hand, it does not let messages accumulate too long so as not to be sent out

  Default: 200ms

• galaxy.talos.producer.max.put.message.number

  The maximum number of message items that Producer sends to a single partition each time a message is taken from the Buffer

  Default: 2000

• galaxy.talos.producer.max.put.message.bytes

  The maximum number of bytes that Producer sends to a single partition each time a message is taken from the Buffer

  Default: 4MB

[Scenario 3] The number of threads whose callback the user wishes to handle is more/less

• galaxy.talos.producer.thread.pool.size

  This configuration is used to specify the number of threads used by the producer to make the messageCallback, that is, the number of threads that execute the callback function after the user sends the message

  Default: 16

[Scenario 4] The user Message does not have a PartitionKey configured and wants to control the Dispatcher

• galaxy.talos.producer.update.partition.id.interval.milli

  When the user does not specify the partitionKey in the Message, the dispatcher in the TalosProducer will be assigned to the currently polled partition id. This configuration specifies the interval for polling the partition id

  Default: 100ms

• galaxy.talos.producer.update.partition.msgnumber

  The Dispatcher polls another dimension of the partition id. If the number of messages pushed to any partition last time exceeds the configured threshold, the partition id polls to the next

  Default: 1000

[Scenario 5] Compression type of user-selected message; Note: In general, no modification is needed to the compression type. Please contact us if necessary

• galaxy.talos.producer.compression.type

  The available types for the compression type of the message are: "NONE", "SNAPPY" and "GZIP"

  Default: SNAPPY