Gitlab runners with Kubernetes in (nearly) no time

Photo by chuttersnap on Unsplash

The gitlab ci/cd toolchain has become a great companion for me in the past years. Since I started using kubernetes more and more I always wanted to utilize the power of kubernetes for the gitlab runners. Furthermore I wanted to get rid of that pesky little badly secured gitlab instance and secure it with TLS (“https”) which became quite necessary later…

As always there is not much to do after you once made it work and invested an hour or two. To spare you the time, let’s get to the details.

Prerequisites

General workflow

  1. Make sure you have the proper certificates at hand: One for the gitlab nginx and (if needed) your local CA.
  2. Create a namespace for your runners.
  3. Insert the certificates as a secret in your kubernetes cluster.
  4. Modify the gitlab-runner yaml file for your needs.
  5. Use the yaml file and helm to install the runners in your kubernetes cluster.

1. Security / Certificates

For obvious reasons it is a good idea to secure the gitlab nginx with TLS. In my case I used TinyCA for local network signing, which can be obtained easily through the package manager of various distributions. But anything that will result in a certificate which works for you will do.

2. Create the namespace

Basically you can be creative here, but seriously do not name it ‘gitlab’ because that can seriously interfere with your routing inside kubernetes!

Here let’s call it runners:

kubectl create namespace runners

3. Create a secret with your certificates

The runners need to be sure that your code repository is exactly what it is supposed to be. We don’t want a man-in-the-middle. So we create a secret with the certificates. This needs at least the certificate from your nginx and in my case I even had to include the CA certificate. (So the crt file contained two BEGIN CERTIFICATE and two END CERTIFICATE blocks.)

kubectl --namespace runners \
  create secret generic gitlab-domain-cert \
  --from-file=gitlab.crt

4. Modify the gitlab runner yaml file

You need to get a configuration file for your runner. (A click on ‘raw’ helps.) Next step is to modify some values:

  • gitlabUrl should be your gitlab instance url including https.
  • runnerRegistrationToken should be the token gitlab provides in its GUI = webpage runner admin settings.
  • certsSecretName has to be the name of our previously set secret: gitlab-domain-cert
  • namespace has to be runners
  • Change RBAC to your needs.
  • (optional) Personally I reduced checkInterval to 5 seconds.

5. Use helm to install your runners

With all the configuration in place, you can initiate the helm magic.

helm install --namespace runners \
  --name gitlab-runner \
  -f gitlab-helm-runner-settings.yml \
  gitlab/gitlab-runner

And you’re done 🙂

Docker in docker (dind) insecure registry

If you are using docker in docker (dind) to create docker containers in a gitlab environment you can use the following snippet to easily use an insecure registry:

dockercreation:
  stage: building
  image: docker:latest

  # When using dind, it's wise to use the overlayfs driver for
  # improved performance.
  variables:
    DOCKER_DRIVER: overlay2

  services:
  - name: docker:dind
    command: ["dockerd-entrypoint.sh", "--insecure-registry=YOUR_HOST:YOUR_PORT"]

  script:
    - echo "create some docker here"

An insecure registry with public access is of course a bad idea …

Erlang poolboy notes

There isn’t much documentation on the great Poolboy worker pool factory. Here are some notes that will hopefully help to understand Poolboy a little bit better.

Notes on the usage

See https://github.com/devinus/poolboy for the original “documentation”. Many will know, but somehow I didn’t: The workers are idle in the pool. One takes them out of the pool and after the work is done, one puts them back in. It’s not like they’re – like I initially thought – working *in* the pool.

(Besides AFAIK checkout and checkin are the only things poolboy utilizes to know whether a worker is busy or not. So you better make sure the checkin is called at least sometime.)

% get a PID of a process that *is* checked in into the pool (=idle)
1> Worker = poolboy:checkout(PoolName).

% now the worker is "checked out" = out of the pool doing work
% => do the actual work here
2> gen_server:call(Worker, Request).
ok

% when you're done => check the worker back in for future use
% you can do this in the worker itself by using the "self" function
3> poolboy:checkin(PoolName, Worker).
ok

Blocking and non-blocking pool worker calls / asynchronous calls

Using “call” like the original usage example is a blocking call. No multicore fun & magic etc. there… For non-blocking consider using “cast” http://erlang.org/doc/man/gen_server.html#cast-2 In some cases it might be the best idea to let the worker do the checkin himself by utilizing the “self” function.

Blocking and non-blocking worker checkout

If you didn’t already do, checkout the source code of poolboy. There you can find – among other things – the full specs of the checkout function. Interestingly enough it has two more optional parameters blocking (default true) and timeout (default 5000).


1> Worker = poolboy:checkout(PoolName).
% returns PID in under 5 seconds or 
full.

2> Worker = poolboy:checkout(PoolName, false).
% no waiting, either you have an idle worker for me or not.


3> Worker = poolboy:checkout(PoolName, true, 10000).
% like the first one but wait 10 secs instead of 5

(At first my idea was to make a merge request for the documentation but the documentation is so straight to the point (for those who basically know how to use it already 😉 ) I didn’t want to ruin that.)

i3 essentials (my notes)

# software
apt-get install nitrogen j4-dmenu-desktop i3 i3lock i3status

# i3 config (~/.config/i3/config)
# i3lock
bindsym Control+Shift+l exec i3lock

# reload the backgrounds you installed with nitrogen earlier
exec --no-startup-id nitrogen --restore

# faster key press detection
exec --no-startup-id xset r rate 200 50

# <a href="http://wiki.netbeans.org/FaqFontRendering#Anti-aliasing_in_NetBeans_7.1">improved netbeans font anti aliasing on lcds</a> / modern screens
# ~./netbeans-8.2/etc/netbeans.conf
# change to
netbeans_default_options="-J-client -J-Xss2m -J-Xms32m -J-Dapple.laf.useScreenMenuBar=true -J-Dapple.awt.graphics.UseQuartz=true -J-Dsun.java2d.noddraw=true -J-Dsun.java2d.dpiaware=true -J-Dsun.zip.disableMemoryMapping=true -J-Dswing.aatext=true -J-Dawt.useSystemAAFontSettings=lcd"

AMD Radeon screen flicker on movement 2015/2016

Since the second to latest kernel update of debian stable my screen flickered in normal xfce4 when there was movement on the screen. I thought it was a bug because i had nothing changed but the recent kernel update did not fix that. It took me some time but if you experience the same you may want to take a look at the dynamic power profiles which according to this

https://wiki.archlinux.org/index.php/ATI#Dynamic_frequency_switching

can lead to flickering when the card changes its frequency. I could not change the value with root so I changed the setting (=disabled) via this boot (grub2) parameter:

radeon.dpm=0

This worked for me.

Easily copy text from your pc to your smartphone (linux)

out

  1. Get qrencode. It’s in a lot of repositories.
  2. In a terminal (http://project-hardware.net is just a text example)
     echo "http://project-hardware.net" | qrencode -o out.png
  3. Open out.png with an image viewer and scan it with your barcode scanner on your smartphone
  4. Done.

Fully sufficient for small texts. For storage limits take a look at: https://en.wikipedia.org/wiki/QR_code#Storage

List all installed daemons on debian,

ubuntu and linux mint which can cost you a quick boot and a responsive system.

trigger@yourhost:~$ dpkg -S /etc/init.d/* | grep -Eo ".*:" | sed "s/://g" | sort | uniq
acpid
alsa-utils
anacron
at
avahi-daemon
bluez
cups-browsed
cups-daemon
dbus
fglrx-atieventsd
hdparm
ifupdown
kbd
kmod
lightdm
lvm2
netfilter-persistent
network-manager
ppp
procps
pulseaudio
rsync
rsyslog
sane-utils
smartmontools
sudo
udev
uml-utilities
util-linux
virtualbox
x11-common

(Basically this command links all files in /etc/init.d/ to their debian package and shows them.)

xbmc show nvidia gpu temp with nouveau

… is a bit tricky but I made it:

<advancedsettings>
 <cputempcommand>sensors|sed -ne "s/Core 1: \+[-+]\([0-9]\+\).*/\1 C/p"</cputempcommand>
 <gputempcommand>sudo nvclock -T | grep -oE "[0-9]{1,}[C,F]" | sed "s/C/ C/g" | sed "s/F/ F/g"</gputempcommand>
 </advancedsettings>

Howto update-safe change Grub2 default to windows

On debian (ubuntu / linux mint should work too) this is quite simple. Grub2 takes menu entry names as default options. This can be done here:

# filename: /etc/default/grub
# search for or add if it not exists:
GRUB_DEFAULT='Windows 7 (loader) (on /dev/sda1)'

Your install might differ. GRUB_DEFAULT takes the menu entries from your grub.cfg usually placed in /boot/grub/

grep -o "menuentry '.*'" /boot/grub/grub.cfg

And here it is:

[...]
menuentry 'Windows 7 (loader) (on /dev/sda1)' --class windows --class os $menuentry_id_option 'osprober-chain-XXXXXXXXXXXXXXXXX'

All what is left to do is update grub as super user:

update-grub

Howto speed up your apt-get like never before

… is quite simple. You get your hands on eatmydata. Yes it’s not a convincing name but hey it’s in the standard repositories of debian and ubuntu so it should do you no harm 🙂

sudo apt-get -y install eatmydata

How to use it

sudo eatmydata apt-get <your regular stuff / install / update /...>

Try it on installs. It is SO MUCH FASTER.

Why the speed? Are there risks?

Normally programs like apt-get are very much power-loss persistent. If there is a sudden power loss you can pretty much resume where you got interrupted. This is possible because apt-get checks often if the data that should be on the disk is really written to it. Old school nerds may know the tool “sync” which is basically what apt-get does a lot.

This is very time consuming because file systems like ext4 and btrfs are not used to being forced to write all the time. They want to keep stuff in buffers and write when they decide it’s time. The is part of the magic speed they can achieve. Eatmydata redirects calls from following programs to sync-like functions programs into the void. Eatmydata makes other programs think they work pretty safe but actually they are not (and hell of a lot faster).

So it’s really fast but you really shouldn’t loose power (or sth. similar) when using eatmydata. But hey my last power failure is years from now and how are the chances you’re using eatmydata in this very moment? I wouldn’t recommend it for cronjobs though.

Update: Problems with Xbian on the Raspberry PI (rpi)

Looks like eatmydata causes problems on the raspberry pi with xbian installed. apt-get / dpkg exits with an error from time to time. But one can always proceed with

sudo dpkg --configure -a

But nevertheless eatmydata is not recommended for the rpi.