docker

I won't say "Long time, no post" - but...

As I had some time at my hands the last couple of months, I was iterating on my idea on hardware optimization using manifest list from the last post Match Node-Specific Needs Using Manifest Lists.

ReCap

The gist is that hardware optimization with containers is a bit of a step back, as the kernel virtualization (aka containers) promises to provide isolation on-top of a Linux (or Windows) kernel without caring to much about the underlying host configuration.

In the previous post I explained how hardware optimized images are used to get the best performance / functionality out of a node.

Containers gain more and more foothold as a lightweight mode of isolating different application relying on kernel features to not spin up emulated hardware - create (rather) heavy virtual machines. That worked great so far, as the resource isolation was only focusing on what the kernel can provide:

• CPU cycles
• Memory
• Input/output to resources controlled by the kernel (e.g. network and filesystems)

The 'Linux Container' workshop at the ISC 2018 was called: High Performance Container Workshop

It was held as part of the International Supercomputing Conference in Frankfurt on June 28nd from 9AM to 6PM at the Marriott Hotel.

I've been working for Docker for a month now and it is already a fun ride. I joined just before the DockerConEU announcement two weeks back, that the Docker Enterprise Edition as well as the Docker Community Editions for Desktops (Docker4Mac/Docker4Win) will support Kubernetes in the future.

Talking to security engineers I was asked how to secure a docker-socket, so that applications like metrics collector, are only able to access a subset of API endpoints.

When looking into it I was looking into the authorisation plugins already out there, but it as far as I understood them, they are only working on TCP sockets and rely on an SSL certificate providing informations about who is accessing them. Recently I tried to create a plugin using the newest plugin system, but that failed to some extend. The plugin system is currently in a transition to be used within the plugin framework and not be directly started at startup.

To circumvent this and get something to work with, I created a little golang tool, that creates a httputil.ReverseProxy, providing a proxy-socket, checking the request against some regular expressions and forwards granted requests to the docker socket on the behalf of the user.

Meet doxy:

Recently Docker released its 17.06 version of the Docker Engine (Announcement) and in there is the new plugin type 'Metrics'.

The example they provide just copies over the internal Prometheus unix socket to an external HTTP endpoint.

ISC 2017 Workshop

The 'Linux Container' workshop at the ISC 2017 was called: Linux Container to optimise IT Infrastructure and High-Performance Workloads.

It was held after the International Supercomputing Conference in Frankfurt on June 22nd from 2PM to 6PM at the Marriott Hotel.

I like the idea and prospect of having only the plain Docker stack running, as it provides a nice experience from development to operations (I am talking about you: DevOps!). I can start with a single container, create a set of (unreplicated) services and try to make it work in a distributed setup - all on my little laptop and stay confident that it will work on a cluster as well.

During the April MeetUp I introduced some guidelines about how to build and run containers, the talk was recorded and is available on youtube:

Yesterdays (ok, late post - at the last) MeetUp was first and foremost about the Container Manifesto, which aims to foster understanding about how to build and run a Container.

Afterwards we figured that I missed 'Containers should start fast (thx Lukasz)' as an additional point - next time. :)

For today I will just put the video in here, a separat blog post might follow - even though I feel it is not that necessary, as no code was executed.

Last week-end I was hacking on my metrics collector to be able to fetch docker-engine and docker-container stats.

But as my MeetUp in Berlin is called M.E.L.I.G. (Metrics,Events,Logs,Inventory and Glue), there is much more to it then metrics.

This week-end I hacked on a new tool: qwatch

Docker 1.13 is on it's way and I like what comes to light.

The highlights from where I stand are:

• service port publishing now as mode host or ingress, which allows for service ports to be outside of the IPVS load-balancer and just exposed on the SWARM node.
• the load-balancer seems to honour established connections
• experimental has an end-point /metrics, which exposes Prometheus formatted metrics.

And this last bit got me interested. So much, though, that I hacked a Prometheus collector into qcollect. :)

I won't apologise for the log delay between posts again, busy times...

But that should discourage you from checking in every once in a while - I got something nice today, at least I think so. :)

A while back I stumbled upon Fullerite, a GOLANG metrics collector, which can reuses the collectors of the python Diamond collector.

One of the issues I had was, that it is not using the event time, but the process time of collected metrics. Thus, if you want to bulk update collected metrics, they will all have the same timestamp of the time they are push to the metrics backend.

Running Docker services on single nodes is quite boring, so let's boot up three boxes with the latest version of docker.

This is a blog post to accompany the talk I gave a the Berlin Docker Meetup:

After a couple of month being busy, it's time for a blog post about Docker Services.

As I stated often - I became a big fan of Consul for service orchestration, service discovery and as a K/V store in my docker stacks.

Since Docker Engine 1.11 the necessary DNS feature to be able to use a 127.0.0.1 address was somewhat kicked, so I had a hard nut to crack. My workaround was to not care about local resolution and use the consul servers as DNS resource. Anyway...

This years 'Linux Container' workshop at the ISC 2016 is called: Docker: Linux Containers to Optimise IT Infrastructure for HPC & BigData.

It was held after the International Supercomputing Conference in Frankfurt on June 23rd at the Marriott hotel.

• Official agenda page: http://www.isc-hpc.com/isc16_ap/

Unlike last year the focus was to provide actionable knowledge about the world of Linux Containers, discuss problems and possible solutions.

After a quick Proof of Concept I pushed it to my little HPC setup with physical nodes and InfiniBand.

Next I aimed to run HPCG within containers instead of a bare-metal run.

Whoo, long time no see (read: blog-post)... :)

But don't you worry guys, I have a nice one this time, I promise.

Even though - CAUTION! - if you have kids in the room, who honour security by not messing around with ssh, be advised that it might be a tough ride...

Talk is done... I hope it brought up some new ideas.

I hope that the videos will be available some day, until then the slides are available here.

Did I mention I love Open-Source software? I do - I really do! :) Using a pull request on Consul, I get closer to a nice setup: github isse

So what I am talking about here? As you readers should already know I use Consul as the backend for my service/node discovery stuff and user related topics.

This blob post goes through the different methods of connecting an HPC cluster in a box (within docker container) so that the network performance is worth the effort and all containers are addressable. It's going to talk about VXLAN, MACLAN and some pipework to glue them together.

Happy new year, y'all!

To kick of 2016 I am going to contribute to FOSDEM by talking about a Multi-host containerised HPC cluster. And since the HPC Advisory Council was so nice to provide an actual HPC resource I will present a 2.0 version of the talk (lessons learned after FOSDEM) at the HPC Advisory Council Workshop in Stanford.

Even though I like Consul a lot (it is the foundation of my stacks in terms of service/node discovery) it's most likely not a replacement for a monitoring framework with notification handlers, distributed checks and a nice dashboard.

I assume that most of the readers have used NAGIOS at some point and decide to hate-love it. It works, but only kinda... :)

Since I like to play around with Docker volumes one day, I have to get used to CEPH somehow. :)

I started by creating a single container that hosts all ceph-daemons needed and which pushes the necessary information to consuls key/value store: qnib/ceph-mono.

A little stack to demonstrate it could be find - as usual - in my stack repository:

In last weeks 3rd M.E.L.I.G. MeetUp I talked about ChatOps, which consolidates communication channels and helps to provide a more natural way of communicating in general.

Why we can’t have nice things

I was asked twice recently how I would transform the stacks I am using into a of-the-shelf Docker HPC cluster.

For starters I will go with a pretty minimalistic approach of leveraging the blog post about docker networking I did and expand it on physical machines.

In this post we will spin up a kvstore holding Consul and connect two distinct docker-machines to the Consul cluster to share the networking configuration.

This post will spin up a SWARM cluster backed by a single Consul container on a separat docker-machine.

Today was the first instance of a M.E.L.I.G. MeetUp. The topic was about metrics in general (broad general) and what the new version of InfluxDB is going to introduce; mainly in terms of the storage backend.

We were kindly hosted by InnoQ (they have a podcast). Thanks for that...

ISC 2015 Workshop

The 'Linux Container' workshop at the ISC 2015 was called:
Docker: User-Friendly Application & Service Containers for HPC Environments.

As promised in my last post here's a blog post about the QNIBTerminal powered SLURM stack with auto generated dashboards. I started writing it two weeks ago, embarrassing - sorry for the delay. As a reminder I'll keep the date.

The stack looks like this:

For those following my blog most of the stack should look familiar.

Since I was ask on hub.docker.com if my qnib/elk image is going to provide kibana4 in the near future I figured it would be worth to blog about it.

The image in question is quite nice for trying the ELK stack out and I take some pride in stating that it's the number 2 image popping up if you search for 'elasticsearch' (and rank by stars). :)

Since I was asked (thanks Dmitry) via mail how to setup QNIBTerminal to run MPI jobs, I created a REAMDE within the qnib/compute repository, but why not put it in a blog post (README.md is Markdown, my blog is Markdown...)?

If you are looking for an excuse to use logstash your local webserver is low hanging fruit.

Someone accesses your website and your web server will store some details about the visit:

10.10.0.1 - - [29/Oct/2014:18:42:18 +0100] "GET / HTTP/1.1" 200 2740 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4"
10.10.0.1 - - [29/Oct/2014:18:42:19 +0100] "GET /css/main.css HTTP/1.1" 200 2805 "http://qnib.org/" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4"
10.10.0.1 - - [29/Oct/2014:18:42:19 +0100] "GET /pics/second_strike_trans.png HTTP/1.1" 200 29636 "http://qnib.org/" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4"