Recently Docker released its 17.06 version of the Docker Engine (Announcement) and in there is the new plugin type 'Metrics'.

The example they provide just copies over the internal Prometheus unix socket to an external HTTP endpoint.

go func() {
            io.Copy(proxyConn, conn)
            conn.(*net.TCPConn).CloseRead()
            proxyConn.(*net.UnixConn).CloseWrite()
        }()
        go func() {
            io.Copy(conn, proxyConn)
            proxyConn.(*net.UnixConn).CloseRead()
            conn.(*net.TCPConn).CloseWrite()
        }()

Let's give it a spin...

$ docker plugin install --grant-all-permissions cpuguy83/docker-metrics-plugin-test:latest
docker run -ti -e SKIP_ENTRYPOINTS=true --network host qnib/plain-elasticsearch bash
[II] qnib/init-plain script v0.4.27
> execute CMD as user 'elasticsearch'
elasticsearch@moby:/$ curl -s http://127.0.0.1:19393/metrics |head
# HELP builder_builds_failed_total Number of failed image builds
# TYPE builder_builds_failed_total counter
builder_builds_failed_total{reason="build_canceled"} 0
builder_builds_failed_total{reason="build_target_not_reachable_error"} 0
builder_builds_failed_total{reason="command_not_supported_error"} 0
builder_builds_failed_total{reason="dockerfile_empty_error"} 0
builder_builds_failed_total{reason="dockerfile_syntax_error"} 0
builder_builds_failed_total{reason="error_processing_commands_error"} 0
builder_builds_failed_total{reason="missing_onbuild_arguments_error"} 0
builder_builds_failed_total{reason="unknown_instruction_error"} 0

OpenTSDB Metrics Plugin

I feel the vibe in Prometheus, but for some things I like the traditional push model more, as the endpoint does not know when a potential collector will scrape the endpoint.

<UPDATE>
I had a discussion with a colleague about the word traditional there, which got me thinking (thanks for the brain-challenge Stephan!).
There are a ton of blog post about it (Prometheus and the Debate Over ‘Push’ Versus ‘Pull’ Monitoring, Pull doesn't scale - or does it? and a nice video from SREcon about monitoring at scale) and I get that it is nice to monitor stuff using the pull model.
I am afraid I have to put my perspective into a distinct blog post some time. In short:

  • Pull if we talk about long lived, regular, static metrics
  • Push in case the metric might go away quickly, you want to be explicit about the timestamp, ....

</UPDATE>

For the internal metrics this might not be super important (long lived, regular, static metric), but maybe some day container metrics will also be available.

Anyhow, I created an OpenTSDB plugin: qnib/docker-plugin-metrics-opentsdb

This plugin connects to the metrics.sock provided by the plugin system and transforms the metrics to OpenTSDB(v1) format, pushing it to a given endpoint.

InfluxDB Stack

To be able to throw the metrics somewhere, an InfluxDB stack with enable OpenTSDB endpoint will do the trick.

$ cat docker-compose.yml
version: '3'
services:
  backend:
    image: qnib/plain-influxdb
    environment:
     - INFLUXDB_META_LOGGING=true
     - INFLUXDB_OPENTSDB_ENABLED=true
    ports:
     - 4242:4242
     - 8083:8083
     - 8086:8086

  frontend:
    image: qnib/plain-grafana4
    ports:
     - 3000:3000
    environment:
     - INFLUXDB_DB=opentsdb
     - INFLUXDB_HOST=tasks.backend
$ docker stack deploy -c docker-compose.yml influxdb
Creating service influxdb_backend
Creating service influxdb_frontend
$

Install, configure and enable plugin

Now we install the plugin.

$ docker plugin install --disable --grant-all-permissions qnib/docker-plugin-metrics-opentsdb
latest: Pulling from qnib/docker-plugin-metrics-opentsdb
9b5a4c6dd405: Download complete
Digest: sha256:e80586adb32cedfb9e5c2a68ef61d96926f76f212d4659efd7c50eac42ee48c5
Status: Downloaded newer image for qnib/docker-plugin-metrics-opentsdb:latest
Installed plugin qnib/docker-plugin-metrics-opentsdb
$ docker plugin ls
ID                  NAME                                         DESCRIPTION                          ENABLED
4cf98db8588c        cpuguy83/docker-metrics-plugin-test:latest   prometheus collector plugin          true
c9ff581daaba        qnib/docker-plugin-metrics-opentsdb:latest   Plugin to push metrics to OpenTSDB   false

The configuration can be changed to adjust the expected OpenTSDB endpoint.

{
    "Env": [{
      "Description": "OpenTSDB host address to send metric to",
      "Name": "OPENTSDB_HOST",
      "Settable": ["value"],
      "Value": "127.0.0.1"
      },{
      "Description": "OpenTSDB port address to send metric to",
      "Name": "OPENTSDB_PORT",
      "Settable": ["value"],
      "Value": "4242"
    },{
      "Description": "Prints OpenTSDB strings to logs, instead of sending it off",
      "Name": "DRY_RUN",
      "Settable": ["value"],
      "Value": "false"
    }]
}

They are set via docker plugin set qnib/docker-plugin-metrics-opentsdb:latest OPENTSDB_HOST=127.0.0.1 and the plugin is started using the enable subcommand.

$  docker plugin enable qnib/docker-plugin-metrics-opentsdb:latest
qnib/docker-plugin-metrics-opentsdb:latest
$ docker plugin ls
ID                  NAME                                         DESCRIPTION                          ENABLED
4cf98db8588c        cpuguy83/docker-metrics-plugin-test:latest   prometheus collector plugin          true
c9ff581daaba        qnib/docker-plugin-metrics-opentsdb:latest   Plugin to push metrics to OpenTSDB   true
$

Grafana

Having Grafana in the docker-compose.yml stack, you can reach an examplary dashboard under localhost:3000 (admin/admin).

That's it for this post... Enjoy and feel free to suggest other plugins. :)