OpenTSDB Docker Metrics Plugin
Recently Docker released its 17.06 version of the Docker Engine (Announcement) and in there is the new plugin type 'Metrics'.
The example they provide just copies over the internal Prometheus unix socket to an external HTTP endpoint.
go func() {
io.Copy(proxyConn, conn)
conn.(*net.TCPConn).CloseRead()
proxyConn.(*net.UnixConn).CloseWrite()
}()
go func() {
io.Copy(conn, proxyConn)
proxyConn.(*net.UnixConn).CloseRead()
conn.(*net.TCPConn).CloseWrite()
}()
Let's give it a spin...
$ docker plugin install --grant-all-permissions cpuguy83/docker-metrics-plugin-test:latest
docker run -ti -e SKIP_ENTRYPOINTS=true --network host qnib/plain-elasticsearch bash
[II] qnib/init-plain script v0.4.27
> execute CMD as user 'elasticsearch'
elasticsearch@moby:/$ curl -s http://127.0.0.1:19393/metrics |head
# HELP builder_builds_failed_total Number of failed image builds
# TYPE builder_builds_failed_total counter
builder_builds_failed_total{reason="build_canceled"} 0
builder_builds_failed_total{reason="build_target_not_reachable_error"} 0
builder_builds_failed_total{reason="command_not_supported_error"} 0
builder_builds_failed_total{reason="dockerfile_empty_error"} 0
builder_builds_failed_total{reason="dockerfile_syntax_error"} 0
builder_builds_failed_total{reason="error_processing_commands_error"} 0
builder_builds_failed_total{reason="missing_onbuild_arguments_error"} 0
builder_builds_failed_total{reason="unknown_instruction_error"} 0
OpenTSDB Metrics Plugin
I feel the vibe in Prometheus, but for some things I like the traditional push model more, as the endpoint does not know when a potential collector will scrape the endpoint.
<UPDATE>
I had a discussion with a colleague about the word traditional there, which got me thinking (thanks for the brain-challenge Stephan!).
There are a ton of blog post about it (Prometheus and the Debate Over ‘Push’ Versus ‘Pull’ Monitoring, Pull doesn't scale - or does it? and a nice video from SREcon about monitoring at scale) and I get that it is nice to monitor stuff using the pull model.
I am afraid I have to put my perspective into a distinct blog post some time. In short:
- Pull if we talk about long lived, regular, static metrics
- Push in case the metric might go away quickly, you want to be explicit about the timestamp, ....
</UPDATE>
For the internal metrics this might not be super important (long lived, regular, static metric), but maybe some day container metrics will also be available.
Anyhow, I created an OpenTSDB plugin: qnib/docker-plugin-metrics-opentsdb
This plugin connects to the metrics.sock
provided by the plugin system and transforms the metrics to OpenTSDB(v1) format, pushing it to a given endpoint.
InfluxDB Stack
To be able to throw the metrics somewhere, an InfluxDB stack with enable OpenTSDB endpoint will do the trick.
$ cat docker-compose.yml
version: '3'
services:
backend:
image: qnib/plain-influxdb
environment:
- INFLUXDB_META_LOGGING=true
- INFLUXDB_OPENTSDB_ENABLED=true
ports:
- 4242:4242
- 8083:8083
- 8086:8086
frontend:
image: qnib/plain-grafana4
ports:
- 3000:3000
environment:
- INFLUXDB_DB=opentsdb
- INFLUXDB_HOST=tasks.backend
$ docker stack deploy -c docker-compose.yml influxdb
Creating service influxdb_backend
Creating service influxdb_frontend
$
Install, configure and enable plugin
Now we install the plugin.
$ docker plugin install --disable --grant-all-permissions qnib/docker-plugin-metrics-opentsdb
latest: Pulling from qnib/docker-plugin-metrics-opentsdb
9b5a4c6dd405: Download complete
Digest: sha256:e80586adb32cedfb9e5c2a68ef61d96926f76f212d4659efd7c50eac42ee48c5
Status: Downloaded newer image for qnib/docker-plugin-metrics-opentsdb:latest
Installed plugin qnib/docker-plugin-metrics-opentsdb
$ docker plugin ls
ID NAME DESCRIPTION ENABLED
4cf98db8588c cpuguy83/docker-metrics-plugin-test:latest prometheus collector plugin true
c9ff581daaba qnib/docker-plugin-metrics-opentsdb:latest Plugin to push metrics to OpenTSDB false
The configuration can be changed to adjust the expected OpenTSDB endpoint.
{
"Env": [{
"Description": "OpenTSDB host address to send metric to",
"Name": "OPENTSDB_HOST",
"Settable": ["value"],
"Value": "127.0.0.1"
},{
"Description": "OpenTSDB port address to send metric to",
"Name": "OPENTSDB_PORT",
"Settable": ["value"],
"Value": "4242"
},{
"Description": "Prints OpenTSDB strings to logs, instead of sending it off",
"Name": "DRY_RUN",
"Settable": ["value"],
"Value": "false"
}]
}
They are set via docker plugin set qnib/docker-plugin-metrics-opentsdb:latest OPENTSDB_HOST=127.0.0.1
and the plugin is started using the enable
subcommand.
$ docker plugin enable qnib/docker-plugin-metrics-opentsdb:latest
qnib/docker-plugin-metrics-opentsdb:latest
$ docker plugin ls
ID NAME DESCRIPTION ENABLED
4cf98db8588c cpuguy83/docker-metrics-plugin-test:latest prometheus collector plugin true
c9ff581daaba qnib/docker-plugin-metrics-opentsdb:latest Plugin to push metrics to OpenTSDB true
$
Grafana
Having Grafana in the docker-compose.yml
stack, you can reach an examplary dashboard under localhost:3000 (admin/admin).
That's it for this post... Enjoy and feel free to suggest other plugins. :)