Kafka

gnmic supports exporting subscription updates to multiple Apache Kafka brokers/clusters simultaneously

A Kafka output can be defined using the below format in gnmic config file under outputs section:

outputs:
  output1:
    # required
    type: kafka 
    # kafka client name. 
    # if left empty, this field is populated with the output name used as output ID (output1 in this example).
    # the full name will be '$(name)-kafka-prod'.
    # If the flag --instance-name is not empty, the full name will be '$(instance-name)-$(name)-kafka-prod.
    # note that each kafka worker (producer) will get client name=$name-$index
    name: ""
    # Comma separated brokers addresses
    address: localhost:9092 
    # Kafka topic name
    topic: telemetry 
    # Kafka SASL configuration
    sasl:
      # SASL user name
      user:
      # SASL password
      password:
      # SASL mechanism: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER are supported
      mechanism:
      # token url for OAUTHBEARER SASL mechanism
      token-url:
    # The total number of times to retry sending a message
    max-retry: 2 
    # Kafka connection timeout
    timeout: 5s 
    # Wait time to reestablish the kafka producer connection after a failure
    recovery-wait-time: 10s 
    # Exported msg format, json, protojson, prototext, proto, event
    format: event 
    # Number of kafka producers to be created 
    num-workers: 1 
    # (bool) enable debug
    debug: false 
    # (int) number of messages to buffer before being picked up by the workers
    buffer-size: 0 
    # (bool) enables the collection and export (via prometheus) of output specific metrics
    enable-metrics: false 
    # list of processors to apply on the message before writing
    event-processors:

Currently all subscriptions updates (all targets and all subscriptions) are published to the defined topic name

When a Prometheus server is enabled, gnmic kafka output exposes 4 prometheus metrics, 3 Counters and 1 Gauge:

  • number_of_kafka_msgs_sent_success_total: Number of msgs successfully sent by gnmic kafka output. This Counter is labeled with the kafka producerID
  • number_of_written_kafka_bytes_total: Number of bytes written by gnmic kafka output. This Counter is labeled with the kafka producerID
  • number_of_kafka_msgs_sent_fail_total: Number of failed msgs sent by gnmic kafka output. This Counter is labeled with the kafka producerID as well as the failure reason
  • msg_send_duration_ns: gnmic kafka output send duration in nanoseconds. This Gauge is labeled with the kafka producerID