Prometheus output with data replication

The purpose of this deployment is to achieve redundancy, high-availability as well as data replication.

The redundancy and high-availability are guaranteed by deploying a gnmic cluster.

The data replication is achieved using a NATS server acting as both a gnmic input and output.

This deployment example includes a:

The leader election and target distribution is done with the help of a Consul server

Each gnmic instance outputs the streamed gNMI data to NATS, and reads back all the data from the same NATS server (including its own),

This effectively guarantees that each instance holds the data streamed by the whole cluster.

Like in the previous examples, each gnmic instance will also register its Prometheus output service in Consul.

But before doing so, it will attempt to acquire a key lock gnmic/$CLUSTER_NAME/prometheus-output, (use-lock: true)

prom-output:
  type: prometheus
  listen: ":9804"
  service-registration:
    address: consul-agent:8500
    use-lock: true # <===
Since only one instance can hold a lock, only one prometheus output is registered, so only one output is scraped by Prometheus.

Deployment files:

Download the files, update the gnmic config files with the desired subscriptions and targets.

Note

The targets outputs list should include the nats output name

Deploy it with:

sudo docker-compose up -d

Check the NATS Output, NATS Input and Prometheus Output documentation pages for more configuration options