This project lets you configure the OpenTelemetry collector to send your Prometheus-format metrics from Hashicorp Consul to Logz.io.

Configuring OpenTelemetry to send your metrics data to Logz.io

Before you begin, you’ll need:

  • Consul installed on your host
  • Golang
Update Consul configuration

  1. Navigate to the Consul server and create a new file called prometheus.json under the Consul config library at /etc/consul.d.

  2. Add the following telemetry stanza to the prometheus.json:

    {
  "telemetry": {
    "disable_hostname": true,
    "prometheus_retention_time": "72h"
   
  }
}
Check the Consul server’s metrics format

Check that the Consul server’s metrics are in Prometheus format at this endpoint:

http://127.0.0.1:8500/v1/agent/metrics?format=prometheus
Download OpenTelemetry collector

If you already have OpenTelemetry, proceed to the next step.

Create a dedicated directory on your host and download the OpenTelemetry collector that is relevant to the operating system of your host.

After downloading the collector, create a configuration file config.yaml.

Configure the receivers

Open the configuration file and make sure that it states the receivers required for your source:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'consul-metrics'
          scrape_interval: 15s
          metrics_path: "/v1/agent/metrics"
          static_configs:
            - targets: ['localhost:8500']
  hostmetrics:
    collection_interval: 1m
    scrapers:
      cpu: ##All execpt Mac
      disk: ##All execpt Mac
      load:
      filesystem:
      memory:
      network:
      process: ##Linux & Windows
Configure the processors

In the same configuration file, add the following to the processors section:

processors:
  resourcedetection/system:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]
Configure the exporters

In the same configuration file, add the following to the exporters section:

exporters:
  prometheusremotewrite:
    endpoint: https://<<LISTENER-HOST>>:8053
    headers:
      Authorization: Bearer <<PROMETHEUS-METRICS-SHIPPING-TOKEN>>

Replace the placeholders to match your specifics. (They are indicated by the double angle brackets << >>):

  • Replace <<PROMETHEUS-METRICS-SHIPPING-TOKEN>> with a token for the Metrics account you want to ship to.
    Here’s how to look up your Metrics token.
  • Replace <<LISTENER-HOST>> with the Logz.io Listener URL for your region, configured to use port 8052 for http traffic, or port 8053 for https traffic. For example, listener.logz.io if your account is hosted on AWS US East, or listener-nl.logz.io if hosted on Azure West Europe.
Configure the service pipeline

In the service section of the configuration file, add the following configuration

service:
  pipelines:
    metrics:
      receivers: [prometheus,hostmetrics]
      processors: [resourcedetection/system]
      exporters:
        - prometheusremotewrite
  telemetry:
    logs:
      level: "debug"
Start the collector

Run the following command:

<path/to>/otelcol-contrib --config ./config.yaml
  • Replace <path/to> with the path to the directory where you downloaded the collector. If the name of your configuration file is different to config, adjust name in the command accordingly.
Check Logz.io for your metrics

Log in to your Logz.io account and navigate to the current instructions page inside the Logz.io app. Install the pre-built dashboard to enhance the observability of your metrics.

To view the metrics on the main dashboard, log in to your Logz.io Metrics account, and open the Logz.io Metrics tab.