2015

Even though I like Consul a lot (it is the foundation of my stacks in terms of service/node discovery) it's most likely not a replacement for a monitoring framework with notification handlers, distributed checks and a nice dashboard.

I assume that most of the readers have used NAGIOS at some point and decide to hate-love it. It works, but only kinda... :)

Since I like to play around with Docker volumes one day, I have to get used to CEPH somehow. :)

I started by creating a single container that hosts all ceph-daemons needed and which pushes the necessary information to consuls key/value store: qnib/ceph-mono.

A little stack to demonstrate it could be find - as usual - in my stack repository:

In last weeks 3rd M.E.L.I.G. MeetUp I talked about ChatOps, which consolidates communication channels and helps to provide a more natural way of communicating in general.

Why we can’t have nice things

I was asked twice recently how I would transform the stacks I am using into a of-the-shelf Docker HPC cluster.

For starters I will go with a pretty minimalistic approach of leveraging the blog post about docker networking I did and expand it on physical machines.

In this post we will spin up a kvstore holding Consul and connect two distinct docker-machines to the Consul cluster to share the networking configuration.

This post will spin up a SWARM cluster backed by a single Consul container on a separat docker-machine.

The second M.E.L.I.G. MeetUp was about the famous ELK stack (Elasticsearch / Logstash / Kibana).

Again, we were kindly hosted by InnoQ (they have a podcast). Thanks for that...

Today was the first instance of a M.E.L.I.G. MeetUp. The topic was about metrics in general (broad general) and what the new version of InfluxDB is going to introduce; mainly in terms of the storage backend.

We were kindly hosted by InnoQ (they have a podcast). Thanks for that...

ISC 2015 Workshop

The 'Linux Container' workshop at the ISC 2015 was called:
Docker: User-Friendly Application & Service Containers for HPC Environments.

OK guys, since the ELK is quite popular on docker-hub, I was wondering if I could push it one notch further. Turns out I could, please welcome qnib/monster...

git (master) $ git clone https://github.com/ChristianKniep/docker-monster.git
git (master) $ cd docker-monster
docker-monster (master) $ docker-compose up -d
Creating dockermonster_monster_1...
docker-monster (master) $

As I wrote last month the zeromq plugin in 1.4 has an issue. And it's still not usable since this patch is not applied. :(

As promised in my last post here's a blog post about the QNIBTerminal powered SLURM stack with auto generated dashboards. I started writing it two weeks ago, embarrassing - sorry for the delay. As a reminder I'll keep the date.

The stack looks like this:

For those following my blog most of the stack should look familiar.

Since I was ask on hub.docker.com if my qnib/elk image is going to provide kibana4 in the near future I figured it would be worth to blog about it.

The image in question is quite nice for trying the ELK stack out and I take some pride in stating that it's the number 2 image popping up if you search for 'elasticsearch' (and rank by stars). :)

Yesterday the Nordic e-Infrastructure Collaboration Conference (NeIC2015) came to an end.

I talked about QNIBTerminal and what I am working on; connecting dots between metrics (graphite-ecosystem), logs (logstash & friends), inventory (QNIBInventory based on a GraphDB) and SLURM (cluster resource scheduler). I put it up on youtube:

The zeromq filter within logstash 1.4 is not working out as expected for me. I opened an issue on github to cope with that. For now I work around this issue by starting logstash 1.3 as a separate instance and let this version deal with zeromq.

Come again?!

For those of you asking WTF?... :) ZeroMQ is a message library that provides multiple patterns like PUB/SUB, PUSH/PULL and others. I got a use-case in which I want specific log events to be handled outside of logstash. And more outside then firing up the ruby filter. I want to process the event within a external daemon to check some things, update the JSON with additional information (lookup names, routes or alike) and after I am done I push it back into the logstash pipeline.

Since I was asked (thanks Dmitry) via mail how to setup QNIBTerminal to run MPI jobs, I created a REAMDE within the qnib/compute repository, but why not put it in a blog post (README.md is Markdown, my blog is Markdown...)?

Last month I was in Lugano presenting the last little study I conducted. The aim of this study was to check if results of an HPC workload depend on the underlying system.

TL;DR

If you don't like this thing called reading or you need more visual input:

The foundation of QNIBTerminal is an image that holds consul and glues everything together. I used the Easter break to refine my qnib/slurm images - this blog post give a quick intro.

At Locafox we are aiming to rule the world, at least the local-commerce part of the internet. For that we need a solid foundation that enables our developers and operational staff (some say DevOps) to do awesome stuff.

I started to write an article about enhancements within my analytics stack (elk/graphite) and got a little bit verbose about consul. Therefore I decided to put it into a seperate article...

Consul

I had consul on my list for some time, but it was just recently that I gave it a spin. And I must admit I am hooked. It provides a nice set of functionalities that I need to bootstrap...

Let's give a quick ride by starting two containers: server and client

It's been a while but I just recently revisited my elk and graphite stack to deal with log events and metrics.