Consul as a (Docker) Service
After a couple of month being busy, it's time for a blog post about Docker Services.
As I stated often - I became a big fan of Consul for service orchestration, service discovery and as a K/V store in my docker stacks.
Since Docker Engine 1.11 the necessary DNS feature to be able to use a 127.0.0.1 address was somewhat kicked, so I had a hard nut to crack. My workaround was to not care about local resolution and use the consul servers as DNS resource. Anyway...
Hello Docker Service
When trying to apply consul to Docker Services I encountered another problem.
A Docker service name is addressable via the embedded DNS of the Docker engine.
So reaching service A from service B is easy by just issuing $ ping A.
How about Consul
The first attempt might be to just spin up a service with three replicas and tell them to join the service name.
$ docker service create --name consul --replicas=3 --publish=8500:8500 \
-e CONSUL_BOOTSTRAP_EXPECT=3 \
-e CONSUL_SKIP_CURL=true \
-e CONSUL_CLUSTER_IPS=consul \
--network consul-net \
qnib/alpn-consul@sha256:846d2005c527d8b764e985166d8c92fd60b9116ea436787f9d289dbc5f0756f5
bx2nrlhdc86unt2zwzm5ms7hd
The problem (as far as I can tell for now) is, that the members are not able to resolve the cluster mates DNS name consul, as they are all part of it. But how to bootstrap a consul cluster if you can not address the rest of the 'team'?
Blue/Green
My current solution is quite simple. I just spin up a seed-consul cluster.
$ docker service create --name consul-seed --replicas=1 --publish=8501:8500 \
-e CONSUL_BOOTSTRAP_EXPECT=3 \
-e CONSUL_SKIP_CURL=true \
-e CONSUL_CLUSTER_IPS=consul-seed,consul \
--network consul-net \
qnib/alpn-consul@sha256:846d2005c527d8b764e985166d8c92fd60b9116ea436787f9d289dbc5f0756f5
It is suffixed -seed, uses a slightly different port and won't never lift off by itself, as it needs at least three servers to bootstrap. The cluster peers to join are consul-seed and consul.
It creates a DC and lingers around, waiting for more servers to join.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
48fce13c478c qnib/alpn-consul@sha256:846d2005c527d8b764e985166d8c92fd60b9116ea436787f9d289dbc5f0756f5 "/opt/qnib/supervisor" 4 seconds ago Up 2 seconds 8300-8301/tcp, 8400/tcp, 8500/tcp consul-seed.1.89001zsr2luzxqvtlmystiy9m
$ docker exec -ti 48fce13c478c consul members
Node Address Status Type Build Protocol DC
48fce13c478c 10.0.1.3:8301 alive server 0.6.4 2 dc1
$
Now I start the real consul service which uses the same CONSUL_CLUSTER_IPS setting, so it will at least join the consul-seed service, as this one is up and running and reachable.
$ docker service create --name consul --replicas=3 --publish=8500:8500 \
-e CONSUL_BOOTSTRAP_EXPECT=3 \
-e CONSUL_SKIP_CURL=true \
-e CONSUL_CLUSTER_IPS=consul-seed,consul \
--network consul-net \
qnib/alpn-consul@sha256:846d2005c527d8b764e985166d8c92fd60b9116ea436787f9d289dbc5f0756f5
Now I got four servers in my dc.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
98b283409afe qnib/alpn-consul@sha256:846d2005c527d8b764e985166d8c92fd60b9116ea436787f9d289dbc5f0756f5 "/opt/qnib/supervisor" 22 seconds ago Up 19 seconds 8300-8301/tcp, 8400/tcp, 8500/tcp consul.2.39i6ztgsh2yhoefo0w2l9y43m
9493c8528e05 qnib/alpn-consul@sha256:846d2005c527d8b764e985166d8c92fd60b9116ea436787f9d289dbc5f0756f5 "/opt/qnib/supervisor" 22 seconds ago Up 20 seconds 8300-8301/tcp, 8400/tcp, 8500/tcp consul.1.2mdx44un16xonzbncdmyl5kf2
0fe68cddca78 qnib/alpn-consul@sha256:846d2005c527d8b764e985166d8c92fd60b9116ea436787f9d289dbc5f0756f5 "/opt/qnib/supervisor" 22 seconds ago Up 20 seconds 8300-8301/tcp, 8400/tcp, 8500/tcp consul.3.eqx7fubo4dxp8p17bmatyvtxw
48fce13c478c qnib/alpn-consul@sha256:846d2005c527d8b764e985166d8c92fd60b9116ea436787f9d289dbc5f0756f5 "/opt/qnib/supervisor" 3 minutes ago Up 3 minutes 8300-8301/tcp, 8400/tcp, 8500/tcp consul-seed.1.89001zsr2luzxqvtlmystiy9m
$ docker exec -ti 48fce13c478c consul members
Node Address Status Type Build Protocol DC
0fe68cddca78 10.0.1.7:8301 alive server 0.6.4 2 dc1
48fce13c478c 10.0.1.3:8301 alive server 0.6.4 2 dc1
9493c8528e05 10.0.1.5:8301 alive server 0.6.4 2 dc1
98b283409afe 10.0.1.6:8301 alive server 0.6.4 2 dc1
$
I tweaked my supervisor parent a bit, so that it forwards the SIGTERM signal in case a container is stopped. This signal is used to gracefully stop a consul container. He simply has trap "consul leave" SIGTERM TERM set in the consul start script and as supervisor passes this down to all services he is going to leave when I kill the no longer needed consul-seed service.
$ docker service rm consul-seed
consul-seed
$ docker exec -ti 98b283409afe consul members
Node Address Status Type Build Protocol DC
0fe68cddca78 10.0.1.7:8301 alive server 0.6.4 2 dc1
48fce13c478c 10.0.1.3:8301 left server 0.6.4 2 dc1
9493c8528e05 10.0.1.5:8301 alive server 0.6.4 2 dc1
98b283409afe 10.0.1.6:8301 alive server 0.6.4 2 dc1
$
Rolling Update
In case I need to update or scale my consul service I will just start the consul-seed service. This is going to be the common entry point which will allow everyone to bond.
$ docker service create --name consul-seed --replicas=1 --publish=8501:8500 \
-e CONSUL_BOOTSTRAP_EXPECT=3 \
-e CONSUL_SKIP_CURL=true \
-e CONSUL_CLUSTER_IPS=consul-seed,consul \
--network consul-net \
qnib/alpn-consul@sha256:846d2005c527d8b764e985166d8c92fd60b9116ea436787f9d289dbc5f0756f5
46u291ex1zx720ob4qk08vcph
$ sleep 30 ; docker exec -ti 98b283409afe consul members
Node Address Status Type Build Protocol DC
0fe68cddca78 10.0.1.7:8301 alive server 0.6.4 2 dc1
139fce7cdb17 10.0.1.3:8301 alive server 0.6.4 2 dc1
48fce13c478c 10.0.1.3:8301 left server 0.6.4 2 dc1
9493c8528e05 10.0.1.5:8301 alive server 0.6.4 2 dc1
98b283409afe 10.0.1.6:8301 alive server 0.6.4 2 dc1
$
No I scale the real service, and kill the seed afterwards.
$ docker service update --replicas=4 consul
consul
$ docker exec -ti 98b283409afe consul members
Node Address Status Type Build Protocol DC
0fe68cddca78 10.0.1.7:8301 alive server 0.6.4 2 dc1
139fce7cdb17 10.0.1.3:8301 alive server 0.6.4 2 dc1
48fce13c478c 10.0.1.3:8301 left server 0.6.4 2 dc1
569a327b4360 10.0.1.8:8301 alive server 0.6.4 2 dc1
9493c8528e05 10.0.1.5:8301 alive server 0.6.4 2 dc1
98b283409afe 10.0.1.6:8301 alive server 0.6.4 2 dc1
$ docker service rm consul-seed
consul-seed
$ docker exec -ti 98b283409afe consul members
Node Address Status Type Build Protocol DC
0fe68cddca78 10.0.1.7:8301 alive server 0.6.4 2 dc1
139fce7cdb17 10.0.1.3:8301 left server 0.6.4 2 dc1
48fce13c478c 10.0.1.3:8301 left server 0.6.4 2 dc1
569a327b4360 10.0.1.8:8301 alive server 0.6.4 2 dc1
9493c8528e05 10.0.1.5:8301 alive server 0.6.4 2 dc1
98b283409afe 10.0.1.6:8301 alive server 0.6.4 2 dc1
$