Kafka Consumer Lag Test

Kafka Consumer lag indicates the lag between Kafka producers and consumers. If the rate of production of data from producers far exceeds the rate at which it is getting consumed by the consumers, then the consumer groups will exhibit a lag. This test helps the administrators to monitor how much lag there is between Kafka producers and consumers.

Target of the test : Apache Kafka Message System

Agent deploying the test : An internal/remote agent

Outputs of the test : One set of results for each consumer group in the target Apache Kafka Message System.

Configurable parameters for the test
Parameter	Description
Test period	How often should the test be executed.
Host	The IP address of the target server that is being monitored.
Port	The port number through which the Apache Kafka Message System communicates. The default port is 9092.
Bootstrap Servers	Bootstrap Servers is a comma-separated list of host and port pairs that are the addresses of the Kafka brokers in a "bootstrap" Kafka cluster that a Kafka client connects to initially to bootstrap itself. By default, the BOOTSTRAP SERVERS text box will display the value as Host:Port. If a single Kafka Broker is associated with multiple Bootstraps in a Kafka cluster, then specify a comma-separated list of values as Host1:Port1, Host2:Port2,.... in this text box. If Kafka Message system is installed in a container, then the value will be displayed as Container ID:Port in this text box.
Lag Percent Cutoff	If the consumer lag percent is more than a specified cut off value, then the consumer information will be displyed in the UI. By default the Lag Percent cutoff value is 25.
Exclude Consumer	The consumer information can be excluded from the UI by specifying the consumer name pattern in the Exclude Consumer text box. By default, the value is none.
DD Frequency	Refers to the frequency with which detailed diagnosis measures are to be generated for this test. The default is 1:1. This indicates that, by default, detailed measures will be generated every time this test runs, and also every time the test detects a problem. You can modify this frequency, if you so desire. Also, if you intend to disable the detailed diagnosis capability for this test, you can do so by specifying none against DD frequency.
Detailed Diagnosis	To make diagnosis more efficient and accurate, the eG Enterprise embeds an optional detailed diagnostic capability. With this capability, the eG agents can be configured to run detailed, more elaborate tests as and when specific problems are detected. To enable the detailed diagnosis capability of this test for a particular server, choose the On option. To disable the capability, click on the Off option. The option to selectively enable/disable the detailed diagnosis capability will be available only if the following conditions are fulfilled: The eG manager license should allow the detailed diagnosis capability Both the normal and abnormal frequencies configured for the detailed diagnosis measures should not be 0.

Measurements made by the test
Measurement	Description	Measurement Unit	Interpretation
Current offset	Indicates the number of current offsets	Number	An offset is a simple integer number that is used by Kafka to maintain the current position of a consumer. If current offset is positive - that means that there is a lag. In most cases, if your Kafka Producer is actively producing messages and the Kafka Consumers are actively consuming, you will have a small lag here. This is expected. The problems start when the lag is significant or is constantly growing. That means that the data is not processed fast enough.
End offset	Indicates the number of end offsets.	Number	Inside the Brokers, data is stored in one or more Topics, and each Topic consists of one or more Partitions. When writing data a Broker actually writes it into a specific Partition. As it writes data it keeps track of the last “write position” in each Partition. This is called Latest Offset, also known as Log End Offset. Each Partition has its own independent Latest Offset.
Lag	Indicates the number of lags between Kafka producers and consumers.	Number	A very low value is required for this measure. Use the detailed diagnosis of this measure to know the Client ID and Host details.
Lag percent	Indicates the percentage of lags between Kafka producers and consumers.	Percent	A very low value is required for this measure.