Skip to content

Prometheus

gnmic supports exposing gnmi updates on a prometheus server, for a prometheus client to scrape.

A Prometheus output can be defined using the below format in gnmic config file under outputs section:

outputs:
  output1:
    type: prometheus # require
    # address to listen on for incoming scrape requests
    listen: :9804 
    # path to query to get the metrics
    path: /metrics 
    # maximum lifetime of metrics in the local cache, #
    # a zero value defaults to 60s, a negative duration (e.g: -1s) disables the expiration
    expiration: 60s 
    # a string to be used as the metric namespace
    metric-prefix: "" 
    # a boolean, if true the subscription name will be appended to the metric name after the prefix
    append-subscription-name: false 
    # boolean, if true the message timestamp is changed to current time
    override-timestamps: false
    # a boolean, enables exporting timestamps received from the gNMI target as part of the metrics
    export-timestamps: false 
    # a boolean, enables setting string type values as prometheus metric labels.
    strings-as-labels: false
    # a boolean, if set to true, the received gNMI notifications are stored in a cache.
    # the prometheus metrics are generated at the time a prometheus server sends scrape request.
    # this behavior allows the processors (if defined) to be run on all the generated events at once.
    # this mode uses more resource compared to the default one, but offers more flexibility when it comes 
    # to manipulating the data to customize the returned metrics using event-processors.
    gnmi-cache: false
    # duration, scrape request timeout.
    # this timer is started when a scrape request is received, 
    # if it is reached, the metrics generation/collection is stopped.
    timeout: 10s
    # enable debug for prometheus output
    debug: false 
    # string, one of `overwrite`, `if-not-present`, ``
    # This field allows populating/changing the value of Prefix.Target in the received message.
    # if set to ``, nothing changes 
    # if set to `overwrite`, the target value is overwritten using the template configured under `target-template`
    # if set to `if-not-present`, the target value is populated only if it is empty, still using the `target-template`
    add-target: 
    # string, a GoTemplate that allow for the customization of the target field in Prefix.Target.
    # it applies only if the previous field `add-target` is not empty.
    # if left empty, it defaults to:
    # {{- if index . "subscription-target" -}}
    # {{ index . "subscription-target" }}
    # {{- else -}}
    # {{ index . "source" | host }}
    # {{- end -}}`
    # which will set the target to the value configured under `subscription.$subscription-name.target` if any,
    # otherwise it will set it to the target name stripped of the port number (if present)
    target-template:
    # list of processors to apply on the message before writing
    event-processors: 
    # Enables Consul service registration
    service-registration:
      # Consul server address, default to localhost:8500
      address:
      # Consul Data center, defaults to dc1
      datacenter: 
      # Consul username, to be used as part of HTTP basicAuth
      username:
      # Consul password, to be used as part of HTTP basicAuth
      password:
      # Consul Token, is used to provide a per-request ACL token which overrides the agent's default token
      token:
      # Prometheus service check interval, for both http and TTL Consul checks,
      # defaults to 5s
      check-interval:
      # Maximum number of failed checks before the service is deleted by Consul
      # defaults to 3
      max-fail:
      # Consul service name
      name:
      # List of tags to be added to the service registration, 
      # if available, the instance-name and cluster-name will be added as tags,
      # in the format: gnmic-instance=$instance-name and gnmic-cluster=$cluster-name
      tags:
      # bool, enables http service check on top of the TTL check
      enable-http-check:
      # string, if enable-http-check is true, this field can be used to specify the http endpoint to be used to the check
      # if provided, this filed with be prepended with 'http://' (if not already present), 
      # and appended with the value in 'path' field.
      # if not specified, it will be derived from the fields 'listen' and 'path'
      http-check-address:
      # if set to true, the gnmic instance will try to ac quire a lock before registering the prometheus output in consul.
      # this allows to register a single instance of the cluster in consul.
      # if the instance which acquired the lock fails, one of the remaining ones will take over.
      use-lock: false

gnmic creates the prometheus metric name and its labels from the subscription name, the gnmic path and the value name.

Metric Generation#

The below diagram shows an example of a prometheus metric generation from a gnmi update

Metric Naming#

The metric name starts with the string configured under metric-prefix.

Then if append-subscription-name is true, the subscription-name as specified in gnmic configuraiton file is appended.

The resulting string is followed by the gNMI path stripped of its keys if there are any.

All non-alphanumeric characters are replaced with an underscore "_"

The 3 strings are then joined with an underscore "_"

If further customization of the metric name is required, the processors can be used to transform the metric name.

For example, a gNMI update from subscription port-stats with path:

/interfaces/interface[name=1/1/1]/subinterfaces/subinterface[index=0]/state/counters/in-octets

is exposed as a metric named:

gnmic_port_stats_interfaces_interface_subinterfaces_subinterface_state_counters_in_octets

Metric Labels#

The metrics labels are generated from the subscription metadata (e.g: subscription-name and source) and the keys present in the gNMI path elements.

For the previous example the labels would be:

{interface_name="1/1/1",subinterface_index=0,source="$routerIP:Port",subscription_name="port-stats"}

Service Registration#

gnmic supports prometheus_output service registration via Consul.

It allows prometheus to dynamically discover new instances of gnmic exposing a prometheus server ready for scraping via its service discovery feature.

If the configuration section service-registration is present under the output definition, gnmic will register the prometheus_output service in Consul.

Configuration Example#

The below configuration, registers a service name gnmic-prom-srv with IP=10.1.1.1 and port=9804

# gnmic.yaml
outputs:
  output1:
    type: prometheus
    listen: 10.1.1.1:9804
    path: /metrics 
    service-registration:
      address: consul-agent.local:8500
      name: gnmic-prom-srv

This allows running multiple instances of gnmic with minimal configuration changes by using prometheus service discovery feature.

Simplified scrape configuration in the prometheus client:

# prometheus.yaml
scrape_configs:
  - job_name: 'gnmic'
    scrape_interval: 10s
    consul_sd_configs:
      - server: $CONSUL_ADDRESS
        services:
          - $service_name

Service Name and ID#

The $service_name to be discovered by prometheus is configured under outputs.$output_name.service-registration.name.

If the service registration name field is not present, the name prometheus-${output_name} will be used.

In both cases the service ID will be prometheus-${output_name}-${instance_name}.

Service Checks#

gnmic registers the service in Consul with a ttl check enabled by default:

  • ttl: gnmic periodically updates the service definition in Consul. The goal is to allow Consul to detect a same instance restarting with a different service name.

If service-registration.enable-http-check is true, an http check is added:

  • http: Consul periodically scrapes the prometheus server endpoint to check its availability.
# gnmic.yaml
outputs:
  output1:
    type: prometheus
    listen: 10.1.1.1:9804
    path: /metrics 
    service-registration:
      address: consul-agent.local:8500
      name: gnmic-prom-srv
      enable-http-check: true

Note that for the http check to work properly, a routable address ( IP or name ) should be specified under listen.

Otherwise, a routable address should be added under service-registration.http-check-address.

Back to top