Regular update of Docker images/containers

After converting my servers into docker setups, I was in need to update the images/containers regularly for security reasons. Baffled I found, that there is no standard update method to make sure that everything is up-to-date.

The ephemeral setup allows you to throw away your containers and images and recreate them with the latest version. As easy as this sounds, you figure that there are some loopholes in the setup.

First we need to understand, there are 3 types of images that we need to keep up-to-date

  1. Images from the docker hub, that just get pulled and are used as they are with some configs
  2. Images from the docker hub, that get pulled and then are only used as base for own dockerfiles
  3. The images created out of own dockerfiles

Ridiculously all three of them need to be updated to make sure everything is up-to-date (e.g., a new build won’t get the latest base image update) and additionally we have to care for cleanup.

I found some solutions in the net to automatically update docker, the so far best version by binfalse.de (Automatically update Docker images). But this leaves out my own dockerfiles with a build and some minor steps, pruning, etc. So I am only using the dupdate script out of the Handy Docker Tools to incorporate in a little script.

So what is the solution

WARNING: This just updates images. If your setup needs additional update steps, you need to plan these in. Otherwise you risk breaking your setup.

You need multiple steps to completely update.

First, I use dupdate to update all docker images coming from a hub, covering the ones I use directly and as base for builds.

/usr/local/sbin/dupdate -v

This will give an error for the images you created out of your own dockerfiles, but update all via pull from docker hub.
Second, I update the images of my own dockerfiles by rebuilding all via docker-compose (If you use docker without docker-compose, you just have to to this for each dockerfile)

/usr/local/bin/docker-compose -f docker-compose.yml build

 

This will use the newly pulled base images in their build, hence create the latest version for your dockerfile.

Now the images are all updated and we only need to restart the containers.

Addition for cleaning up

However you end up with a lot of images tagged or named <none>. These are your old images, which are now cluttering the hard drive.

The ones only tagged with <none> are the ones you updated from docker hub, the ones with name and tag <none> are the ones you build.

You will need to do a image prune to get rid of them and free up space.

/usr/bin/docker image prune -a --force

Warning: This will erase all older images. If you need them as a safety precaution, skip this step.

Advertisements

Reinstall GRUB after BIOS Update on LUKS encrypted system

Due to reasons unknown I upgraded my Lenovos BIOS via a USB Stick. 🙂 Everything went well, however after reboot, all my boot options for Linux Mint were gone.
Turns out, somehow my boot setup was erased as well.

Using UEFI without CSM and without secure boot with LUKS encrypted Linux Mint, this was already an issue when first installing. Getting everything right seems to be more of good luck.

So what is the solution

To get back your boot menu, I tried several things, e.g. boot-repair. However since my system is LUKS encrypted, I guess the tools all had some problems.

To get back my system, I accessed my old system via chroot from a Linux Live CD. In this case Linux Mint Live CD.

1. Boot from Linux Live CD, get keyboard locale and network set up. Unlock your LUKS device to the crypt mapper. The Linux Mint Live CD does that by only clicking on the drive, if you are on console, take a look here: https://wiki.ubuntuusers.de/LUKS/

2. Mount all necessary partitions out of old system
To find the LUKS drives, use lvscan:

sudo lvscan

For me my /root turns out to be in /dev/mint-vg/root
If you have separated partitions, e.g. for home, or other devices, make sure to adapt parts below for mounting.

sudo mount /dev/mint-vg/root /mnt
sudo mount /dev/sda2 /mnt/boot
for i in /dev /dev/pts /proc /sys /run ; do sudo mount -B $i /mnt$i ; done
sudo mount -o bind /etc/resolv.conf /mnt/etc/resolv.conf
sudo mount /dev/sda1 /mnt/boot/efi/

Make sure you add the /mnt/boot/efi, otherwise grub will complain grub-install: “cannot find EFI directory”. It is not included in your boot partition, but a separate partition.

3. Enter the chroot environment

sudo chroot /mnt /bin/bash

4. Install GRUB

sudo grub-install /dev/sda

Just a reboot and the system should work as before.

SendXMPP mail forward on Debian Jessie

To have a more comfortable way of receiving messages by my servers, I wanted all my root E-Mails to be forwarded to my mobile via XMPP. I only have a limited exim4 on my machine running, configured for local mail delivery only.

So what is the solution

1. Install sendxmpp

apt-get install sendxmpp

2. Create config file as “/etc/sendxmpp.conf”

@

as an example:

sendinguser@jabber.mydomain.com supersecretpassword! mydomain.com

3. Set the right permissions and owner

chmod 600 /etc/sendxmpp.conf
chown Debian-exim:Debian-exim /etc/sendxmpp.conf

4. Create a script to call sendxmpp as “/usr/sbin/mail2xmpp”. It might be that you could put this completely into the alias, however I decided to use the script.
Exchange with your receiving ID. “-t” enables the TLS connection for sending the message.

#!/bin/bash 
echo "$(cat)" | sendxmpp -t  -f /etc/sendxmpp.conf

5. Make script executable

chmod 755 /usr/sbin/mail2xmpp

6. Create the alias for the user, which E-Mails you want to forward in “/etc/aliases”

root: , |/usr/sbin/mail2xmpp

7. To activate pipe forwarding we have to create “/etc/exim4/exim4.conf.localmacros”

SYSTEM_ALIASES_PIPE_TRANSPORT = address_pipe

8. Run newaliases and restart exim4 for config to take effect

newaliases
service exim4 restart

Now you should be able to test if it works, by simply sending a local test E-Mail to user “root”.

Unlock LUKS via SSH in Debian

As already described in my previous post (Headless Debian install via SSH), I am dealing with a headless system. As I am encrypting my system and drives with LUKS, I need a way to enter the password in case of a reboot.

So what is the solution

1. Install Dropbear on the server

apt-get install dropbear

2. Configure initramfs network usage; edit “/etc/initramfs-tools/initramfs.conf”. You probably have to add the lines for dropbear and update the device string.

This configuration is using DHCP to obtain an IP, if you have a static configuration, use:
IP=<SERVER-IP>::<STANDARD-GATEWAY>:<SUBNETMASK>:<HOSTNAME>:eth0:off

# 
# DROPBEAR: [ y | n ]
# 
# Use dropbear if available.
# 

DROPBEAR=y
DEVICE=eth0
IP=:::::eth0:dhcp

3. Delete the standard private and public keys on the server

rm /etc/initramfs-tools/root/.ssh/id_rsa
rm /etc/initramfs-tools/root/.ssh/id_rsa.pub

4. Create your own key pair (we assume you use id_rsa as a name) on your client machine and upload it to the server.

ssh-keygen
scp ~/.ssh/id_rsa.pub myuser@debian_headless:id_rsa.pub

5. Then log in to the server and add the key to authorized_key file an remove the public key on the server.

ssh myuser@debian_headless
sudo sh -c "cat id_rsa.pub >> /etc/initramfs-tools/root/.ssh/authorized_keys"
rm id_rsa.pub

6. Now we need to update initramfs and grub

update-initramfs -u -k all
update-grub2

7. On some configurations the network won’t get reconfigured on runtime values, hence we need to trigger an update. Edit “/etc/network/interfaces” and add as first line of the primary interface:

pre-up ip addr flush dev eth0

8. Restart server and log in from your client

ssh -i ~/.ssh/id_rsa root@<server-ip>

9. Set the password to unlock

echo -n "<LUKS encryption password>" > /lib/cryptsetup/passfifo
exit

The server should now boot normally and regular SSH should come up.

Optional: You can also create a little script for the passphrase in “/etc/initramfs-tools/hooks/unlock”

#!/bin/sh
PREREQ=""
 prereqs()
 {
 echo "$PREREQ"
 }
 case $1 in
 prereqs)
 prereqs
 exit 0
 ;;
 esac
. /usr/share/initramfs-tools/hook-functions
cat > "${DESTDIR}/root/unlock" << EOF #!/bin/sh /lib/cryptsetup/askpass 'passphrase: ' > /lib/cryptsetup/passfifo
 EOF
chmod u+x "${DESTDIR}/root/unlock"
exit 0

Do not forget to make it executable

chmod +x /etc/initramfs-tools/hooks/unlock

And update initramfs

update-initramfs -u -k all
update-grub2

Headless Debian install via SSH

Having build a NAS system recently, I realized I do not have any monitors or keyboards at home anymore. Hence installing Debian will be hard.

I looked around and the solution would be a headless install via ssh.

This post is based on some work from S.G. Vulcan’s post Installing Debian using only SSH

His post was a good start, but I only could make it work for a Debian Jessie netinstall image after some changes.

So what is the solution

1. Download the latest netinstall image from Debian, I used “debian-8.3.0-amd64-netinst.iso”

2. Mount the ISO to a folder

mkdir isoorig
mount -o loop -t iso9660 debian-8.3.0-amd64-netinst.iso isoorig

3. Copy to new folder called isonew

mkdir isonew rsync -a -H –exclude=TRANS.TBL isoorig/ isonew

4. Change the menu to load SSH on boot by default, edit isonew/isolinux/txt.cfg

remove:

“menu default” from “label install”

add:
Here I changed the vga parameters and adapted the kernel parameters to arm to match my original ISO.

label netinstall
menu label ^Install Over SSH 
menu default
kernel /install.arm/vmlinuz
append auto=true vga=788 file=/cdrom/preseed.cfg initrd=/install.arm/initrd.gz locale=en_US console-keymaps-at/keymap=us

change:

“default install” to “default netinstall”

5. Create isonew/preseed.cfg file

I adapted the locale and keyboard settings for Germany and added the selection of the keyboard-configuration. This would otherwise be an open question during the install and we won’t reach the SSH startup.
Also I added a check for non-free firmware, which popped up on one of my machines which had wireless.

#### Contents of the preconfiguration file
### Localization
# Locale sets language and country.
d-i debian-installer/locale select de_DE
# Keyboard selection.
d-i console-keymaps-at/keymap select de
d-i keyboard-configuration/xkb-keymap select de
### Network configuration
# netcfg will choose an interface that has link if possible. This makes it
# skip displaying a list if there is more than one interface.
d-i netcfg/choose_interface select auto
# Any hostname and domain names assigned from dhcp take precedence over
# values set here. However, setting the values still prevents the questions
# from being shown, even if values come from dhcp.
d-i netcfg/get_hostname string newdebian
d-i netcfg/get_domain string local
# If non-free firmware is needed for the network or other hardware, you can
# configure the installer to always try to load it, without prompting. Or
# change to false to disable asking.
d-i hw-detect/load_firmware boolean true
# The wacky dhcp hostname that some ISPs use as a password of sorts.
#d-i netcfg/dhcp_hostname string radish
d-i preseed/early_command string anna-install network-console
# Setup ssh password
d-i network-console/password password install
d-i network-console/password-again password install

6. Recreate md5sum.txt
md5sum.txt is read only, so you need to change this. Also I had better luck with creating the md5sum.txt with the changed command below.

chmod 666 md5sum.txt
find -follow -type f -exec md5sum {} \; > md5sum.txt
chmod 444 md5sum.txt

7. Create ISO file to burn

xorriso -as mkisofs -D -r -J -joliet-long -l -V "Debian headless" -b isolinux/isolinux.bin -c isolinux/boot.cat -iso-level 3 -no-emul-boot -partition_offset 16 -boot-load-size 4 -boot-info-table -isohybrid-mbr /usr/lib/syslinux/isohdpfx.bin -o ../debian-8.3.0-amd64-netinst-headless.iso ../isonew

xorriso is creating a correct partition table, which is for some reason not done with mkisofs only. If you do not have it installed use “apt-get install xorriso”.
The original command would work in VMs, maybe even on a cd-rom, however not for USB sticks.

8. Use the ISO

The ISO can be burned to an USB stick and used to boot. It will automatically configure the network with DHCP (yes, you need to have a way to find the IP, e.g. on your router) and start SSH.

The user for the ssh connection is “installer” the password is “install”

Upgrading Spamassassin Debian Wheezy to Jessie

After the upgrade of Wheezy to Jessie, Spamassassin is not added to the startup services. Hence if you were using it before in your mail setup, you will run into the following error in /var/log/syslog

spamc: connect to spamd on ::1 failed, retrying (#1 of 3): Connection refused
spamc: connect to spamd on 127.0.0.1 failed, retrying (#1 of 3): Connection refused

etc.
Debian has a similar bug filed (#764438) even after a fresh reinstall.

So what is the solution

Simply activate Spamassassin via systemctl enable spamassassin

RSyslog simple sorting to files, here iptables

If you have a firewall, you probably have 99% of the entires in your syslog filled with iptables denied info. So there are two options to get rid of that. Turn of the logging of iptables or move it out of the syslog.
As I want to have the logging, I need to move it. You will find a lot of explanations on RSyslog in the official documentation, however I found that the simple case I was looking for was not described.

So I am using the following to create the syslog entries

iptables -I INPUT 5 -m limit –limit 5/min -j LOG –log-prefix “iptables denied: ” –log-level 7

So what is the solution

You might have more logging or similar ones in your configuration. The important part here is the –log-prefix as we will sort via this.
To create a filter, create /etc/rsyslog.d/iptables.conf and add the following

# Proper file permissions
$FileCreateMode 0600

# Ruleset for sorting
if $msg contains ‘iptables denied: ‘ then {
# new destination
/var/log/iptables.log
# stop processing message, otherwise it will carbon to syslog
stop
}

After restarting rsyslogd, this will now divert the iptables entries to the new logging location identified by the prefix.

The stop is particularly important, as the rule above diverts the logging, but without the stop it would still end up in syslog. If you Google setups like this you will often find a ~ instead of the stop. Which is deprecated and cause a warning in the log, however works as well.
Sidenote: Remember to put the stop in brackets for the if/then clause, otherwise you stop all logging to syslog.

As a last step you should create a logrotation for the new log. I did it very simple by creating the file /etc/logrotate.d/iptables and added

/var/log/iptables.log {
rotate 5
monthly
compress
missingok
notifempty
}

Logrotate is started via cron and will read the new config in the next run. Alternatively you can run the config directly via logrotate -f /etc/logrotate.d/iptables