GOCD + metric stacks via Docker Services
I won't apologise for the log delay between posts again, busy times...
But that should discourage you from checking in every once in a while - I got something nice today, at least I think so. :)
Hardware Update
I've been busy looking hard on Docker Services and how to use them to orchestrate stuff. As this week-end in Berlin brought bad weather - hence some time on my plate to revisit my little home-server setup.
I bought a new machine with more oomph to supply my little video-capture setup Shuttle DS57U.
This bugger has a 14nm Celeron instead of the old 22nm Celeron J1900 (Shuttle XS36v4). Even though it has four cores it was sweating catching up with the video decoding. The new one even got encoder/decoder within the Chipset.
Docker Services Server
Anyway, so I was able to kick out my Raspi3 and use the aforementioned XS36v4 to supply my home automation system (based on OpenHABA, but that is for another post).
This weekend I refined my setup, so that I spin up a couple of Docker Services. The beauty is, that Docker Services survive a reboot. The Docker SWARM will come up and will make sure that all services are started. Furthermore I could employ the DS57U to spin up and connect to the SWARM cluster to help out on tasks.
Basic setup
The setup is pretty simple:
- Download your favourite Linux distro and install it on the machine
- Install the latest docker-engine (>1.12)
That is basically it. I will show the little walkthrough creating services on my macbook, but any docker-engine will do.
Create a SWARM
Simply do this...
$ docker swarm init
Swarm initialized: current node (akp0y81cjkoqzbylidgnpwxq3) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join \
--token SWMTKN-1-577ue6408olgvjik6udaerh8qqk7lia75crjb27vr3bfu943ou-eco4gev0hnu0mhnjph34vepam \
192.168.65.2:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
$
That's that.
Create the services
Clone my service-orchestration and service-scripts repository:
git clone https://github.com/qnib/service-orchestration.git
git clone https://github.com/qnib/service-scripts.git
Be aware: Creating a service means that the image has to be downloaded from the interweb. Even though my images are quite small - YMMV.
$ docker images |egrep "voodoo-query with manual removal"
qnib/influxdb latest b14dae6e20ff 5 hours ago 171.6 MB
qnib/d12-alpn-gocd-agent latest 872cb4fb5f16 19 hours ago 858.3 MB
qnib/qcollect latest 0080d849c817 2 weeks ago 153.9 MB
qnib/grafana3 latest f2a16cecc247 3 weeks ago 1.078 GB
qnib/d12-alpn-gocd-agent latest 872cb4fb5f16 19 hours ago 858.3 MB
$
Looking at it, there is some room for improvement - 1GB? Srsly!?...
Docker Application Bundle
I created a little script that turns docker-compose.yml files into docker services as the Docker Application Bundle lacks some features. In the future the script should be unnecessary, but if you try to create a bundle from this:
version: '2'
services:
gocd-server: #1
image: qnib/d12-gocd-server
networks:
- consul-net
ports:
- 8153:8153
environment:
- DC_NAME=swarm
- GOCD_SERVER_CLEAN_WORKSPACE=false
- CONSUL_CLUSTER_IPS=consul
# if more then one service is started, this is going to cause trouble
- CONSUL_NODE_NAME=gocd-server
labels:
- "org.qnib.service.depend_on=consul"
volumes:
- /srv/go-server/serverBackups/:/opt/go-server/artifacts/serverBackups/
- /srv/go-server/pipelines/:/srv/go-server/pipelines/
networks:
consul-net:
external: true
It will complain about all the stuff that is currently not implemented (which is OK, it's an iterative approach and only the first iteration).
$ docker-compose bundle
WARNING: Unsupported top level key 'networks' - ignoring
WARNING: Unsupported key 'volumes' in services.gocd-agent - ignoring
WARNING: Unsupported key 'volumes' in services.gocd-server - ignoring
Wrote bundle to gocd.dab
$ jq . gocd.dab
{
"Services": {
"gocd-server": {
"Env": [
"DC_NAME=swarm",
"GOCD_SERVER_CLEAN_WORKSPACE=false",
"CONSUL_CLUSTER_IPS=consul",
"CONSUL_NODE_NAME=gocd-server"
],
"Image": "qnib/d12-gocd-server@sha256:0548acc49bafe03c0fba1a91620ca6f734a37c85aa3de1b2b7fd28ffa1a2f03e",
"Labels": {
"org.qnib.service.depend_on": "consul"
},
"Networks": [
"consul-net"
],
"Ports": [
{
"Port": 8153,
"Protocol": "tcp"
}
]
}
},
"Version": "0.1"
}
It at least misses out on the volumes and the compose-file has not yet the notion of replication level. Therefore I will stick with a script that turns a docker compose file into a docker service create command.
Consul
By firing up the start-service.sh script, it will transform the YAML into JSON (easier to use with jq), parse some stuff and assemble a docker service create command.
$ cd ../consul
$ bash ../../service-scripts/service-scripts/swarm/bin/start-service.sh docker-compose.yaml
>> COMPOSE_FILE=docker-compose.yaml
>> COMPOSE_JSON=docker-compose.json
>> Looking into Service 'consul', expected scale '1'
Error: no such service: consul
>>> Create consul-net: docker network create -d overlay consul-net
99if8j509wjvvlq9d38bbhup4
jq: error (at docker-compose.json:27): Cannot iterate over null (null)
[PROCEED] No service running...
>>> docker service create --name consul -e DC_NAME=swarm -e CONSUL_BOOTSTRAP_EXPECT=1 -e CONSUL_SKIP_CURL=true -e CONSUL_NODE_NAME=consul \
--mode=replicated --replicas=1 --network consul-net \
--publish 8500:8500 \
--mount type=bind,source=/etc/hostname,target=/etc/docker-hostname \
\
qnib/alpn-consul:d12
4fkvajoecqzs0r3x3qjgzm69g
$
After a couple of seconds the consul service will be up'n'running and reachable under localhost:8500 (YMMV depending on the IP address of your docker-engine).
InfluxDB
I updated my InfluxDB stack to use Alpine and the latest version.
$ cd ../influxdb
$ bash ../../service-scripts/service-scripts/swarm/bin/start-service.sh docker-compose.yaml
>> COMPOSE_FILE=docker-compose.yaml
>> COMPOSE_JSON=docker-compose.json
>> Looking into Service 'influxdb', expected scale '1'
Error: no such service: influxdb
>>> Network consul-net already existing...
jq: error (at docker-compose.json:26): Cannot iterate over null (null)
[PROCEED] Service dependency 'consul' is running, off we go..
[PROCEED] No service running...
>>> docker service create --name influxdb -e DC_NAME=swarm -e CONSUL_NODE_NAME=influxdb -e COLLECT_METRICS=false -e INFLUXDB_DATABASES=qcollect -e CONSUL_CLUSTER_IPS=consul -e INFLUXDB_META_LOGGING=true \
--mode=replicated --replicas=1 --network consul-net \
--publish 2003:2003 --publish 8083:8083 --publish 8086:8086 \
\
\
qnib/influxdb
3f9dsaa0hwzldxkuz4dvg5pg3
$ docker service ls
ID NAME REPLICAS IMAGE COMMAND
3f9dsaa0hwzl influxdb 0/1 qnib/influxdb
4fkvajoecqzs consul 1/1 qnib/alpn-consul:d12
$
After a couple of seconds you should be able to reach localhost:8083.
Make sure to change the Database in the upper right to qcollect.
qcollect
What is a metrics back-end without some metrics? Therefore, we spin up qcollect - my fullerite fork.
$ cd ../qcollect
$ bash ../../service-scripts/service-scripts/swarm/bin/start-service.sh docker-compose.yaml
>> COMPOSE_FILE=docker-compose.yaml
>> COMPOSE_JSON=docker-compose.json
>> Looking into Service 'qcollect', expected scale 'global'
Error: no such service: qcollect
>>> Network consul-net already existing...
jq: error (at docker-compose.json:26): Cannot iterate over null (null)
[PROCEED] Service dependency 'influxdb' is running, off we go..
[PROCEED] No service running...
>>> docker service create --name qcollect -e CONSUL_DC_NAME=swarm -e DC_NAME=swarm -e QCOLLECT_INFLUXDB_ENABLED=true -e QCOLLECT_INFLUXDB_SERVER=influxdb -e DOCKER_HOST=unix:///var/run/docker.sock -e QCOLLECT_INTERVAL=2 -e CONSUL_CLUSTER_IPS=consul -e CONSUL_NODE_NAME=qcollect \
--mode global --network consul-net \
\
--mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \
\
qnib/qcollect
cona63mc69s339gy2brsbkpwi
$ docker service ls
ID NAME REPLICAS IMAGE COMMAND
3f9dsaa0hwzl influxdb 1/1 qnib/influxdb
4fkvajoecqzs consul 1/1 qnib/alpn-consul:d12
cona63mc69s3 qcollect global qnib/qcollect
$
If we now wait a sec' - there will be metrics flowing in...
Grafana3
OK, but metrics to look at is nothing without something to visualise it.
$ cd ../grafana3
$ bash ../../service-scripts/service-scripts/swarm/bin/start-service.sh docker-compose.yaml
>> COMPOSE_FILE=docker-compose.yaml
>> COMPOSE_JSON=docker-compose.json
>> Looking into Service 'grafana3', expected scale '1'
Error: no such service: grafana3
>>> Network consul-net already existing...
jq: error (at docker-compose.json:22): Cannot iterate over null (null)
[PROCEED] Service dependency 'consul' is running, off we go..
[PROCEED] Service dependency 'influxdb' is running, off we go..
[PROCEED] No service running...
>>> docker service create --name grafana3 -e DC_NAME=swarm -e COLLECT_METRICS=false -e CONSUL_CLUSTER_IPS=consul -e CONSUL_NODE_NAME=grafana3 \
--mode=replicated --replicas=1 --network consul-net \
--publish 3000:3000 \
\
\
qnib/grafana3
67dsx4pa9byj9a6l9p3qf56jx
$ docker service ls
ID NAME REPLICAS IMAGE COMMAND
3f9dsaa0hwzl influxdb 1/1 qnib/influxdb
4fkvajoecqzs consul 1/1 qnib/alpn-consul:d12
67dsx4pa9byj grafana3 0/1 qnib/grafana3
cona63mc69s3 qcollect global qnib/qcollect
$
After it is downloaded (that's true for the rest of the services as well), that is...
$ docker service ps grafana3
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR
50zhx77p00and3835dm0z4z3u grafana3.1 qnib/grafana3 moby Running Preparing 55 seconds ago
$
Afterwards just open localhost:3000, login with admin/admin and open the predefined dashboard.
GoCD
Last I need something to free me from keeping track of what I have build and how - a Continuous Integration platform like GoCD will do. This one defines two services and will start them both.
$ cd ../gocd
$ bash ../../service-scripts/service-scripts/swarm/bin/start-service.sh docker-compose.yaml
>> COMPOSE_FILE=docker-compose.yaml
>> COMPOSE_JSON=docker-compose.json
>> Looking into Service 'gocd-server', expected scale '1'
Error: no such service: gocd-server
>>> Network consul-net already existing...
[PROCEED] Service dependency 'consul' is running, off we go..
[PROCEED] No service running...
>>> docker service create --name gocd-server -e DC_NAME=swarm -e GOCD_SERVER_CLEAN_WORKSPACE=false -e CONSUL_CLUSTER_IPS=consul -e CONSUL_NODE_NAME=gocd-server \
--mode=replicated --replicas=1 --network consul-net \
--publish 8153:8153 \
\
\
qnib/d12-gocd-server
8l5v3tzroi2t03qc4zrzhvzh0
>> Looking into Service 'gocd-agent', expected scale 'global'
Error: no such service: gocd-agent
>>> Network consul-net already existing...
jq: error (at docker-compose.json:49): Cannot iterate over null (null)
[PROCEED] Service dependency 'consul' is running, off we go..
[PROCEED] Service dependency 'gocd-server' is running, off we go..
[PROCEED] No service running...
>>> docker service create --name gocd-agent -e DC_NAME=swarm -e GO_SERVER=gocd-server -e GOCD_LOCAL_DOCKERENGINE=false -e CONSUL_CLUSTER_IPS=consul \
--mode global --network consul-net --network consul-net \
\
--mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \
\
qnib/d12-alpn-gocd-agent
8smpngduu56jx4fxwj4znluxr
$
Btw, a recently realised that $(docker ps -ql) comes in handy, as it will show you the container_id of the last started container.
$ docker service ps gocd-server
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR
1aiq3s15anidplqyz70qyy0aq gocd-server.1 qnib/gocd-server:d12 moby Running Starting 4 seconds ago
$ docker inspect $(docker ps -ql) |jq ".[] |.Config.Healthcheck"{
"Test": [
"CMD-SHELL",
"/opt/qnib/gocd/server/bin/healthcheck.sh"
],
"Interval": 5000000000,
"Timeout": 2000000000,
"Retries": 120
}
$
As the GoCD server is a heavy java8 process (I am waiting for the day, when everyone comes fresh from university and fancies something less heavy [duck'n'cover]).
But once, it is up hit localhost:8153 to get the first screen...
How to set it up with the stack - that's another post (as said before).
Enjoy!
The end
That's it, the services should hum along and even survive a reboot, as the containers are stop at shutdown and just started by SWARM when the system comes up again. :)
$ docker service ls git:(master↑3|✚1
ID NAME REPLICAS IMAGE COMMAND
3f9dsaa0hwzl influxdb 1/1 qnib/influxdb
4fkvajoecqzs consul 1/1 qnib/alpn-consul:d12
67dsx4pa9byj grafana3 1/1 qnib/grafana3
8l5v3tzroi2t gocd-server 1/1 qnib/d12-gocd-server
8smpngduu56j gocd-agent global qnib/d12-alpn-gocd-agent
cona63mc69s3 qcollect global qnib/qcollect
$
So long Christian