Docker 1.13 Prometheus end-point and qcollect
Docker 1.13 is on it's way and I like what comes to light.
The highlights from where I stand are:
- service port publishing now as mode
host
oringress
, which allows for service ports to be outside of the IPVS load-balancer and just exposed on the SWARM node. - the load-balancer seems to honour established connections
- experimental has an end-point
/metrics
, which exposes Prometheus formatted metrics.
And this last bit got me interested. So much, though, that I hacked a Prometheus collector into qcollect. :)
Docker 1.13 experimental Vagrant Stack
I created a docker-1.13 vagrant stack with three small VMs.
$ git clone https://github.com/qnib/vagrant-orchestration.git
$ cd vagrant-orchestration
$ vagrant up swarm0 git:(master|✚1
Bringing machine 'swarm0' up with 'virtualbox' provider...
==> swarm0: Importing base box 'bento/ubuntu-16.04'...
==> swarm0: Matching MAC address for NAT networking...
*snip*
==> swarm0: Created symlink from /etc/systemd/system/multi-user.target.wants/qcollect.service to /lib/systemd/system/qcollect.service.
$
After that a SWARM cluster with only one node as formed.
$ export DOCKER_HOST=192.168.100.10:2376
$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
y54euwa9prtmvdfilxkb7fkds * swarm0 Ready Active Leader
$
Not only instantiates the Vagrant provisioning the SWARM cluster, but also a metrics stack...
$ docker service ls
ID NAME REPLICAS IMAGE COMMAND
b6kbrioq3paq influxdb 1/1 qnib/influxdb@sha256:8f560be674b3601194de4c80c126e90a55916983bc076bf9ae5c944378d123b5
dw31bqwdk9hu http global qnib/httpcheck@sha256:59a2d0f3d2b31e83056f9290e05708ce5a78434a894953ed9f0b141b39c222b1
ig3b76a2ukp0 grafana3 1/1 qnib/alpn-grafana3@sha256:60e8c2cbe6edd438c210a89e7354f2cabd5cda0f94aba9f1ec25765772e1e7dc
u3371h61eu1b consul 1/1 qnib/alpn-consul@sha256:e9a518ec4fd39fbcaf4f47192f20c03728f7050cb3f69cc40c1c57934a9ea6e8
$
... and qcollect-as-a-service...
$ vagrant ssh swarm0 -- sudo systemctl status qcollect git:(master|✚1…
* qcollect.service - QCollect metrics collector
Loaded: loaded (/lib/systemd/system/qcollect.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2016-11-20 16:34:28 UTC; 20min ago
Main PID: 6124 (qcollect)
Tasks: 7
Memory: 8.0M
CPU: 3.991s
CGroup: /system.slice/qcollect.service
`-6124 /usr/local/bin/qcollect
Nov 20 16:54:25 swarm0 qcollect[6124]: time="2016-11-20T16:54:25.623326252Z" level=info msg="POST of 100 metrics to InfluxDB took 0.005613 seconds" app=qcollect handler=InfluxDB pkg=handler
Nov 20 16:54:28 swarm0 qcollect[6124]: time="2016-11-20T16:54:28.625531659Z" level=info msg="POST of 26 metrics to InfluxDB took 0.012559 seconds" app=qcollect handler=InfluxDB pkg=handler
Nov 20 16:54:28 swarm0 qcollect[6124]: time="2016-11-20T16:54:28.62863199Z" level=info msg="POST of 60 metrics to InfluxDB took 0.012831 seconds" app=qcollect handler=InfluxDB pkg=handler
Nov 20 16:54:30 swarm0 qcollect[6124]: time="2016-11-20T16:54:30.629644927Z" level=info msg="POST of 100 metrics to InfluxDB took 0.009841 seconds" app=qcollect handler=InfluxDB pkg=handler
Nov 20 16:54:33 swarm0 qcollect[6124]: time="2016-11-20T16:54:33.626255199Z" level=info msg="POST of 26 metrics to InfluxDB took 0.009733 seconds" app=qcollect handler=InfluxDB pkg=handler
Nov 20 16:54:33 swarm0 qcollect[6124]: time="2016-11-20T16:54:33.62657149Z" level=info msg="POST of 60 metrics to InfluxDB took 0.009258 seconds" app=qcollect handler=InfluxDB pkg=handler
Nov 20 16:54:35 swarm0 qcollect[6124]: time="2016-11-20T16:54:35.625379582Z" level=info msg="POST of 100 metrics to InfluxDB took 0.008441 seconds" app=qcollect handler=InfluxDB pkg=handler
Nov 20 16:54:38 swarm0 qcollect[6124]: time="2016-11-20T16:54:38.623678086Z" level=info msg="POST of 26 metrics to InfluxDB took 0.010786 seconds" app=qcollect handler=InfluxDB pkg=handler
Nov 20 16:54:38 swarm0 qcollect[6124]: time="2016-11-20T16:54:38.627324732Z" level=info msg="POST of 60 metrics to InfluxDB took 0.012158 seconds" app=qcollect handler=InfluxDB pkg=handler
Nov 20 16:54:40 swarm0 qcollect[6124]: time="2016-11-20T16:54:40.635768324Z" level=info msg="POST of 100 metrics to InfluxDB took 0.007869 seconds" app=qcollect handler=InfluxDB pkg=handler
Grafana3 can now be reached under http://192.168.100.10:3000/dashboard/db/docker-engine.
After starting the second and third node...
vagrant up swarm1 swarm2 git:(master|✚1…
Bringing machine 'swarm1' up with 'virtualbox' provider...
Bringing machine 'swarm2' up with 'virtualbox' provider...
==> swarm1: Importing base box 'bento/ubuntu-16.04'...
*snip*
==> swarm2: This node joined a swarm as a manager.
==> swarm2: Created symlink from /etc/systemd/system/multi-user.target.wants/qcollect.service to /lib/systemd/system/qcollect.service.
... the dashboard shows stats from all three engines and the global service http
, spans across the complete cluster.
That is all as an initial step...