Encrypting More: /boot Joins The Party

Typically when installing major linux distros they make it easy to select encryption as an option to have encrypted block devices. This is great! The not so great part is the linux kernel and the initial ramdisk aren't typically invited to the party; they are left sitting in a separate and unencrypted /boot partition. Historically it has been necessary to leave /boot unencrypted because bootloaders didn't support decrypting block devices. However, there are some dangers to leaving the bootloader and ramdisks unencrypted (see this post).

Newer versions of GRUB do support booting from encrypted block devices (a reference here). This means that we can theoretically boot from a device that is encrypted. And the theory is right!

While the installers don't make it easy to actually install in this setup (without a separate boot partition) it is actually pretty easy to convert an existing system to use this setup. I'll step through doing this on a Fedora 22 system (I have done this one Fedora 21 in the past).

The typical disk configuration (with crypto selected) from a vanilla install of Fedora 22 looks like this:

[root@localhost ~]# lsblk -i -o NAME,TYPE,MOUNTPOINT
NAME                                          TYPE  MOUNTPOINT
sda                                           disk  
|-sda1                                        part  /boot
`-sda2                                        part  
  `-luks-cb85c654-7561-48a3-9806-f8bbceaf3973 crypt 
    |-fedora-swap                             lvm   [SWAP]
    `-fedora-root                             lvm   /

What we need to do is copy the files from the /boot partition and into the /boot directory on the root filesystem. We can do this easily with a bind mount like so:

[root@localhost ~]# mount --bind / /mnt/
[root@localhost ~]# cp -a /boot/* /mnt/boot/
[root@localhost ~]# cp -a /boot/.vmlinuz-* /mnt/boot/
[root@localhost ~]# diff -ur /boot/ /mnt/boot/
[root@localhost ~]# umount /mnt 

This copied the files over and verified the contents matched. The final step is to unmount the partition and to remove the mount from /etc/fstab. Since we'll no longer be using that partition we don't want kernel updates to be written to the wrong place:

[root@localhost ~]# umount /boot
[root@localhost ~]# sed -i -e '/\/boot/d' /etc/fstab

The next step is to write out a new grub.cfg that loads the appropriate modules for loading from the encrypted disk:

[root@localhost ~]# cp /boot/grub2/grub.cfg /boot/grub2/grub.cfg.backup
[root@localhost ~]# grub2-mkconfig > /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.0.4-301.fc22.x86_64
Found initrd image: /boot/initramfs-4.0.4-301.fc22.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-3f9d22f02d854d9a857066570127584a
Found initrd image: /boot/initramfs-0-rescue-3f9d22f02d854d9a857066570127584a.img
done
[root@localhost ~]# cat /boot/grub2/grub.cfg | grep cryptodisk
        insmod cryptodisk
        insmod cryptodisk

And finally we need to reinstall the GRUB bootloader with GRUB_ENABLE_CRYPTODISK=y set in /etc/default/grub:

[root@localhost ~]# echo GRUB_ENABLE_CRYPTODISK=y >> /etc/default/grub
[root@localhost ~]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora/swap rd.lvm.lv=fedora/root rd.luks.uuid=luks-cb85c654-7561-48a3-9806-f8bbceaf3973 rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_CRYPTODISK=y
[root@localhost ~]# grub2-install /dev/sda 
Installing for i386-pc platform.
Installation finished. No error reported.

After a reboot you now get your grub prompt:

image

Unfortunately this does mean that you have to type your password twice on boot but at least your system is more encrypted than it was before. This may not completely get rid of the attack vector described in this post as there is still part of the bootloader that isn't encrypted, but at least the grub stage2 and the kernel/ramdisk are encrypted and should make it much harder to attack.

Happy Encrypting!

Dusty

Atomic Host Red Hat Summit Lab

Red Hat Summit was a blast this year. I participated in several Hands On Labs to help the community learn about the new tools that are available in the ecosystem. For one of the labs I wrote up a section on Atomic Host, but more specifically on rpm-ostree. I have copied a portion of the lab here as well as added example text to the code blocks.

Lab Intro

Atomic Host is a minimalistic operating system that is designed to contain a very small subset of tools that are needed for running container based applications. A few of it's features are shown below:

  • It is Lightweight
    • a small base means less potential issues.
  • Provides Atomic Upgrades and Rollbacks
    • upgrades/rollbacks are staged and take effect on reboot
  • Static and Dynamic
    • software/binaries in /usr and other similar directories are read-only
      • this guarantees no changes have been made to the software
    • configuration and temporary directories are read/write
      • you can still make important configuration changes and have them propagate forward

We will explore some of these features as we illustrate a bit of the lifecycle of managing a RHEL Atomic Host.

Hello rpm-ostree World

In an rpm-ostree world you can't install new software on the system or even touch most of the software that exists. Go ahead and try:

-bash-4.2# echo 'Crazy Talk' > /usr/bin/docker
-bash: /usr/bin/docker: Read-only file system

What we can do is configure the existing software on the system using the provided mechanisms for configuration. We can illustrate this by writing to motd and then logging in to see the message:

-bash-4.2# echo 'Lab 1 is fun' > /etc/motd
-bash-4.2# ssh root@localhost
Last login: Fri Jun  5 02:26:59 2015 from localhost
Lab 1 is fun
-bash-4.2# exit
logout
Connection to localhost closed.

Even though we can't install new software, your Atomic Host operating system isn't just a black box. The rpm command is there and we can run queries just the same as if we were on a traditional system. This is quite useful because we can use the tools we are familiar with to investigate the system. Try out a few rpm queries on the Atomic Host:

-bash-4.2# rpm -q kernel
kernel-3.10.0-229.4.2.el7.x86_64
-bash-4.2# rpm -qf /usr/bin/vi
vim-minimal-7.4.160-1.el7.x86_64
-bash-4.2# rpm -q --changelog util-linux | wc -l
1832

Another nice thing about atomic, or rather the underlying ostree software is that it is like git for your OS. At any point in time you can see what has changed between what was delivered in the tree vs. what is on the system. That means for those few directories that are read/write, you can easily view what changes have been made to them.

Let's take a look at the existing differences between what we have and what was delivered in the tree:

-bash-4.2# ostree admin config-diff | head -n 5
M    adjtime
M    motd
M    group
M    hosts
M    gshadow

You can see right in the middle the the motd file we just modified.

As a final step before we do an upgrade let's run a container and verify all is working:

-bash-4.2# docker run -d -p 80:80 --name=test
repo.atomic.lab:5000/apache
e18a5f7d54c8dbe0d352e2c2854af16d27f166d11b95bc37a3b4267cfcd39cd6
-bash-4.2# curl http://localhost
Apache
-bash-4.2# docker rm -f test
test

Performing an Upgrade

Ok, now that we have took a little tour, let's actually perform an upgrade in which we move from one version of the tree to a newer version. First, let's check the current status of the system:

-bash-4.2# atomic host status
  TIMESTAMP (UTC)         VERSION   ID             OSNAME               REFSPEC
* 2015-05-30 04:10:40               d306dcf255     rhel-atomic-host     lab:labtree
  2015-05-07 19:00:48     7.1.2     203dd666d3     rhel-atomic-host     rhel-atomic-host-ostree:rhel-...

Note that the * indicates which tree is currently booted. The ID is a short commit ID for that commit in the tree. The REFSPEC for the latest tree specifies the remote we are using (lab) and the ref that we are tracking (labtree). Quite a lot of information!

A fun fact is that the atomic host command is just a frontend for the rpm-ostree utility. It has some of the functionality of the rpm-ostree utility that is suitable for most daily use. Let's use rpm-ostree now to check the status:

-bash-4.2# rpm-ostree status
  TIMESTAMP (UTC)         VERSION   ID             OSNAME               REFSPEC
* 2015-05-30 04:10:40               d306dcf255     rhel-atomic-host     lab:labtree
  2015-05-07 19:00:48     7.1.2     203dd666d3     rhel-atomic-host     rhel-atomic-host-ostree:rhel-...

The next step is to actually move to a new tree. For the purposes of this lab, and to illustrate Atomic's usefulness, we are actually going to upgrade to a tree that has some bad software in it. If we were to run an atomic host upgrade command then it would actually take us to the newest commit in the repo. In this case we want to go to an intermediate commit (a bad one) so we are going to run a special command to get us there:

-bash-4.2# rpm-ostree rebase lab:badtree

26 metadata, 37 content objects fetched; 101802 KiB transferred in 7 seconds
Copying /etc changes: 26 modified, 8 removed, 70 added
Transaction complete; bootconfig swap: yes deployment count change: 0
Freed objects: 180.1 MB
Deleting ref 'lab:labtree'
Changed:
  etcd-2.0.11-2.el7.x86_64
  kubernetes-0.17.1-1.el7.x86_64
Removed:
  setools-console-3.3.7-46.el7.x86_64

What we did there was rebase to another ref (badtree), but we kept with the same remote (lab).

So we have rebased to a new tree but we aren't yet using that tree. During upgrade the new environment is staged for the next boot, but not yet being used. This allows the upgrade to be atomic. Before we reboot we can check the status. You will see the new tree as well as the old tree listed. The * still should be next to the old tree since that is the tree that is currently booted and running:

-bash-4.2# atomic host status
  TIMESTAMP (UTC)         ID             OSNAME               REFSPEC
  2015-05-30 04:39:22     146b72d9d7     rhel-atomic-host     lab:badtree
* 2015-05-30 04:10:40     d306dcf255     rhel-atomic-host     lab:labtree

After checking the status reboot the machine in order to boot into the new tree.

Rolling Back

So why would you ever need to roll back? It's a perfect world and nothing ever breaks right? No! Sometimes problems arise and it is always nice to have an undo button to fix it. In the case of Atomic, there is atomic host rollback. Do we need to use it now? Let's see if everything is OK on the system:

-bash-4.2# atomic host status
  TIMESTAMP (UTC)         ID             OSNAME               REFSPEC
* 2015-05-30 04:39:22     146b72d9d7     rhel-atomic-host     lab:badtree
  2015-05-30 04:10:40     d306dcf255     rhel-atomic-host     lab:labtree
-bash-4.2# 
-bash-4.2# docker run -d -p 80:80 --name=test repo.atomic.lab:5000/apache
ERROR
-bash-4.2# curl http://localhost
curl: (7) Failed connect to localhost:80; Connection refused
-bash-4.2# systemctl --failed | head -n 3
UNIT           LOAD   ACTIVE SUB    DESCRIPTION
docker.service loaded failed failed Docker Application Container Engine

Did anything fail? Of course it did. So let's press the eject button and get ourselves back to safety:

-bash-4.2# atomic host rollback
Moving 'd306dcf255b370e5702206d064f2ca2e24d1ebf648924d52a2e00229d5b08365.0' to be first deployment
Transaction complete; bootconfig swap: yes deployment count change: 0
Changed:
  etcd-2.0.9-2.el7.x86_64
  kubernetes-0.15.0-0.4.git0ea87e4.el7.x86_64
Added:
  setools-console-3.3.7-46.el7.x86_64
Sucessfully reset deployment order; run "systemctl reboot" to start a reboot
-bash-4.2# reboot

Now, let's check to see if we are back to a good state:

-bash-4.2# atomic host status
  TIMESTAMP (UTC)         ID             OSNAME               REFSPEC
* 2015-05-30 04:10:40     d306dcf255     rhel-atomic-host     lab:labtree
  2015-05-30 04:39:22     146b72d9d7     rhel-atomic-host     lab:badtree
-bash-4.2# docker run -d -p 80:80 --name=test repo.atomic.lab:5000/apache
a28a5f80bc2d1da9d405199f88951a62a7c4c125484d30fbb6eb2c4c032ef7f3
-bash-4.2# curl http://localhost
Apache
-bash-4.2# docker rm -f test
test

All dandy!

Final Upgrade

So since the badtree has been released the developers fixed the bug and have put out a new tree that is fixed. Now we can upgrade to the newest tree. As part of this upgrade let's explore some of the rpm-ostree features.

First, create a file in /etc/ and show that ostree knows that it has been created and differs from the tree that was delivered:

-bash-4.2# echo "Before Upgrade d306dcf255" > /etc/before-upgrade.txt
-bash-4.2# ostree admin config-diff | grep before-upgrade
A    before-upgrade.txt

Now we can do the upgrade:

-bash-4.2# atomic host upgrade --reboot
Updating from: lab:labtree

48 metadata, 54 content objects fetched; 109056 KiB transferred in 9 seconds
Copying /etc changes: 26 modified, 8 removed, 74 added
Transaction complete; bootconfig swap: yes deployment count change: 0

After the upgrade let's actually run a few commands to see the actual difference is (in terms of rpms) between the two trees:

-bash-4.2# atomic host status
  TIMESTAMP (UTC)         ID             OSNAME               REFSPEC
* 2015-05-30 05:12:55     ec89f90273     rhel-atomic-host     lab:labtree
  2015-05-30 04:10:40     d306dcf255     rhel-atomic-host     lab:labtree
-bash-4.2# rpm-ostree db diff -F diff d306dcf255 ec89f90273
ostree diff commit old: d306dcf255 (d306dcf255b370e5702206d064f2ca2e24d1ebf648924d52a2e00229d5b08365)
ostree diff commit new: ec89f90273 (ec89f902734e70b4e8fbe5000e87dd944a3c95ffdb04ef92f364e5aaab049813)
!atomic-0-0.22.git5b2fa8d.el7.x86_64
=atomic-0-0.26.gitcc9aed4.el7.x86_64
!docker-1.6.0-11.el7.x86_64
=docker-1.6.0-15.el7.x86_64
!docker-python-1.0.0-35.el7.x86_64
=docker-python-1.0.0-39.el7.x86_64
!docker-selinux-1.6.0-11.el7.x86_64
=docker-selinux-1.6.0-15.el7.x86_64
!docker-storage-setup-0.0.4-2.el7.noarch
=docker-storage-setup-0.5-2.el7.x86_64
!etcd-2.0.9-2.el7.x86_64
=etcd-2.0.11-2.el7.x86_64
!kubernetes-0.15.0-0.4.git0ea87e4.el7.x86_64
=kubernetes-0.17.1-4.el7.x86_64
+kubernetes-master-0.17.1-4.el7.x86_64
+kubernetes-node-0.17.1-4.el7.x86_64
!python-websocket-client-0.14.1-78.el7.noarch
=python-websocket-client-0.14.1-82.el7.noarch
-setools-console-3.3.7-46.el7.x86_64

This shows added, removed, changed rpms between the two trees.

Now remember that file we created before the upgrade? Is it still there? Let's check and also create a new file that represents the after upgrade state:

-bash-4.2# cat /etc/before-upgrade.txt
Before Upgrade d306dcf255
-bash-4.2# echo "After Upgrade ec89f90273" > /etc/after-upgrade.txt
-bash-4.2# cat /etc/after-upgrade.txt
After Upgrade ec89f90273

Now which of the files do you think will exist after a rollback? Only you can find out!:

-bash-4.2# rpm-ostree rollback --reboot 
Moving 'd306dcf255b370e5702206d064f2ca2e24d1ebf648924d52a2e00229d5b08365.0' to be first deployment
Transaction complete; bootconfig swap: yes deployment count change: 0

After rollback:

-bash-4.2# atomic host status
  TIMESTAMP (UTC)         ID             OSNAME               REFSPEC         
* 2015-05-30 04:10:40     d306dcf255     rhel-atomic-host     lab:labtree     
  2015-05-30 05:12:55     ec89f90273     rhel-atomic-host     lab:labtree     
-bash-4.2# ls -l /etc/*.txt
-rw-r--r--. 1 root root 26 Jun  5 03:35 /etc/before-upgrade.txt

Fin!

Now you know quite a bit about upgrading, rolling back, and querying information from your Atomic Host. Have fun exploring!

Dusty

Fedora 22 Updates-Testing Atomic Tree

It has generally been difficult to test new updates for the rpm-ostree or ostree packages for Atomic Host. This is because in the past you had to build your own tree in order to test them. Now, however, Fedora has starting building a tree based off the updates-testing yum repositories. This means that you can easily test updates by simply running Fedora Atomic Host and rebasing to the fedora-atomic/f22/x86_64/testing/docker-host ref:

# rpm-ostree rebase fedora-atomic:fedora-atomic/f22/x86_64/testing/docker-host
# reboot

After reboot you are now (hopefully) booted into the tree with updates baked in. You can do your tests and report your results back upstream. If you ever want to go back to following the stable branch then you can do that by running:

# rpm-ostree rebase fedora-atomic:fedora-atomic/f22/x86_64/docker-host
# reboot

Testing updates this way can apply to any of the packages within Atomic Host. Since Atomic Host has a small footprint the package you want to test might not be included, but if it is then this is a great way to test things out.

Dusty

Fedora 22 Now Swimming in DigitalOcean

cross posted from this fedora magazine post

DigitalOcean is a cloud provider that provides a one-click deployment of a Fedora Cloud instance to an all-SSD server in under a minute. After some quick work by the DigitalOcean and Fedora Cloud teams we are pleased to announce that you can now make it rain Fedora 22 droplets!

One significant change over previous Fedora droplets is that this is the first release to have support for managing your kernel internally. Meaning if you dnf update kernel-core and reboot then you'll actually be running the kernel you updated to. Win!

Here are a couple more tips for Fedora 22 Droplets:

  • Like with other DigitalOcean images, you will log in with your ssh key as root rather than the typical fedora user that you may be familiar with when logging in to a Fedora cloud image.
  • Similar to Fedora 21, Fedora 22 also has SELinux enabled by default.
  • Fedora 22 should be available in all the newest datacenters in each region, but some legacy datacenters aren't supported. If you have a problem you think is Fedora specific then drop us an email at , ping us in #fedora-cloud on freenode, or visit the Fedora cloud trac to see if it is already being worked on.

Visit the DigitalOcean Fedora landing page and spin one up today!

Happy Developing!
Dusty

F22 Cloud/Atomic Test Day May 7th!

Hey everyone! Fedora 22 is on the cusp of being released and the Fedora Cloud Working Group has elected to organize a test day for May 7th in order to work out some bugs before shipping it off to the rest of the world.

With a new release comes some new features and tools. We are working on Vagrant images as well as a testing tool called Tunir. Joe Brockmeier has a nice writeup about Vagrant and Kushal Das maintains some docs on Tunir.

On the test day we will be testing both the Cloud Base Image and the Fedora Atomic Host cloud image. The landing pages where we are organizing instructions and information are here (for Cloud Base) and here (for Atomic). If you're available to test on the test day (or any other time) please go there and fill out your name and test results.

Happy Testing!

Dusty

Crisis Averted.. I’m using Atomic Host

This blog has been running on Docker on Fedora 21 Atomic Host since early January. Occasionally I log in and run rpm-ostree upgrade followed by a subsequent reboot (usually after I inspect a few things). Today I happened to do just that and what did I come up with?? A bunch of 404s. Digging through some of the logs for the systemd unit file I use to start my wordpress container I found this:

systemd[1]: wordpress-server.service: main process exited, code=exited, status=1/FAILURE
docker[2321]: time="2015-01-31T19:09:24-05:00" level="fatal" msg="Error response from daemon: Cannot start container 51a2b8c45bbee564a61bcbffaee5bc78357de97cdd38918418026c26ae40fb09: write /sys/fs/cgroup/memory/system.slice/docker-51a2b8c45bbee564a61bcbffaee5bc78357de97cdd38918418026c26ae40fb09.scope/memory.memsw.limit_in_bytes: invalid argument"

Hmmm.. So that means I have updated to the latest atomic and docker doesn't work?? What am I to do?

Well, the nice thing about atomic host is that in moments like these you can easily go back to the state you were before you upgraded. A quick rpm-ostree rollback and my blog was back up and running in minutes.

Whew! Crisis averted.. But now what? Well the nice thing about atomic host is that I can easily go to another (non-production) system and test out exactly the same scenario as the upgrade that I performed in production. Some quick googling led me to this github issue which looks like it has to do with setting memory limits when you start a container using later versions of systemd.

Let's test out that theory by recreating this failure.

Recreating the Failure

To recreate I decided to start with the Fedora 21 atomic cloud image that was released in December. Here is what I have:

-bash-4.3# ostree admin status
* fedora-atomic ba7ee9475c462c9265517ab1e5fb548524c01a71709539bbe744e5fdccf6288b.0
    origin refspec: fedora-atomic:fedora-atomic/f21/x86_64/docker-host
-bash-4.3#
-bash-4.3# rpm-ostree status
  TIMESTAMP (UTC)         ID             OSNAME            REFSPEC
* 2014-12-03 01:30:09     ba7ee9475c     fedora-atomic     fedora-atomic:fedora-atomic/f21/x86_64/docker-host
-bash-4.3#
-bash-4.3# rpm -q docker-io systemd
docker-io-1.3.2-2.fc21.x86_64
systemd-216-12.fc21.x86_64
-bash-4.3#
-bash-4.3# docker run --rm --memory 500M busybox echo "I'm Alive"
Unable to find image 'busybox' locally
Pulling repository busybox
4986bf8c1536: Download complete 
511136ea3c5a: Download complete 
df7546f9f060: Download complete 
ea13149945cb: Download complete 
Status: Downloaded newer image for busybox:latest
I'm Alive

So the system is up and running and able to run a container with the --memory option set. Now lets upgrade to the same commit that I did when I saw the failure earlier and reboot:

-bash-4.3# ostree pull fedora-atomic 153f577dc4b039e53abebd7c13de6dfafe0fb64b4fdc2f5382bdf59214ba7acb

778 metadata, 4374 content objects fetched; 174535 KiB transferred in 156 seconds
-bash-4.3#
-bash-4.3# echo 153f577dc4b039e53abebd7c13de6dfafe0fb64b4fdc2f5382bdf59214ba7acb > /ostree/repo/refs/remotes/fedora-atomic/fedora-atomic/f21/x86_64/docker-host
-bash-4.3#
-bash-4.3# ostree admin deploy fedora-atomic:fedora-atomic/f21/x86_64/docker-host
Copying /etc changes: 26 modified, 4 removed, 36 added
Transaction complete; bootconfig swap: yes deployment count change: 1
-bash-4.3#
-bash-4.3# ostree admin status
  fedora-atomic 153f577dc4b039e53abebd7c13de6dfafe0fb64b4fdc2f5382bdf59214ba7acb.0
    origin refspec: fedora-atomic:fedora-atomic/f21/x86_64/docker-host
* fedora-atomic ba7ee9475c462c9265517ab1e5fb548524c01a71709539bbe744e5fdccf6288b.0
    origin refspec: fedora-atomic:fedora-atomic/f21/x86_64/docker-host
-bash-4.3# 
-bash-4.3# rpm-ostree status
  TIMESTAMP (UTC)         ID             OSNAME            REFSPEC
  2015-01-31 21:08:35     153f577dc4     fedora-atomic     fedora-atomic:fedora-atomic/f21/x86_64/docker-host
* 2014-12-03 01:30:09     ba7ee9475c     fedora-atomic     fedora-atomic:fedora-atomic/f21/x86_64/docker-host
-bash-4.3# reboot

Note that I had to manually update the ref to point to the commit I downloaded in order to get this to work. I'm not sure why this is but it wouldn't work otherwise.

Ok now I had a system using the same tree that I was when I saw the failure. Let's check to see if it still happens:

-bash-4.3# rpm-ostree status
  TIMESTAMP (UTC)         ID             OSNAME            REFSPEC
* 2015-01-31 21:08:35     153f577dc4     fedora-atomic     fedora-atomic:fedora-atomic/f21/x86_64/docker-host
  2014-12-03 01:30:09     ba7ee9475c     fedora-atomic     fedora-atomic:fedora-atomic/f21/x86_64/docker-host
-bash-4.3#
-bash-4.3# rpm -q docker-io systemd
docker-io-1.4.1-5.fc21.x86_64
systemd-216-17.fc21.x86_64
-bash-4.3#
-bash-4.3# docker run --rm --memory 500M busybox echo "I'm Alive"
FATA[0003] Error response from daemon: Cannot start container d79629bfddc7833497b612e2b6d4cc2542ce9a8c2253d39ace4434bbd385185b: write /sys/fs/cgroup/memory/system.slice/docker-d79629bfddc7833497b612e2b6d4cc2542ce9a8c2253d39ace4434bbd385185b.scope/memory.memsw.limit_in_bytes: invalid argument

Yep! Looks like it consistently happens. This is good because this is a recreator that can now be used by anyone to verify the problem on their own. For completeness I'll go ahead and rollback the system to show that the problem goes away when back in the old state:

-bash-4.3# rpm-ostree rollback 
Moving 'ba7ee9475c462c9265517ab1e5fb548524c01a71709539bbe744e5fdccf6288b.0' to be first deployment
Transaction complete; bootconfig swap: yes deployment count change: 0
Changed:
  NetworkManager-1:0.9.10.0-13.git20140704.fc21.x86_64
  NetworkManager-glib-1:0.9.10.0-13.git20140704.fc21.x86_64
  ...
  ...
Removed:
  flannel-0.2.0-1.fc21.x86_64
Sucessfully reset deployment order; run "systemctl reboot" to start a reboot
-bash-4.3# reboot

And the final test:

-bash-4.3# rpm-ostree status
  TIMESTAMP (UTC)         ID             OSNAME            REFSPEC
* 2014-12-03 01:30:09     ba7ee9475c     fedora-atomic     fedora-atomic:fedora-atomic/f21/x86_64/docker-host
  2015-01-31 21:08:35     153f577dc4     fedora-atomic     fedora-atomic:fedora-atomic/f21/x86_64/docker-host
-bash-4.3# docker run --rm --memory 500M busybox echo "I'm Alive"
I'm Alive

Bliss! And you can thank Atomic Host for that.

Dusty

quick audit rules for sanity check

Most of the time when I really want to figure out what is going on deep within a piece of software I break out strace and capture all the gory detail. Unfortunately it isn't always that easy to manipulate and run something from the command line but I have found that some simple uses of the audit daemon can give you great insight without having to dig too deep.

Example Problem

I have a script, switch.py, I want to call via a bound key sequence from i3 window manager. However, I notice that nothing happens when I press the key sequence. Is the script failing or is the script not getting called at all? auditd and auditctl can help us figure this out.

Using audit

To take advantage of system auditing the daemon must be up and running:

# systemctl status auditd.service | grep active
       Active: active (running) since Sun 2015-01-25 13:56:27 EST; 1 day 9h ago

You can then add a watch for read/write/execute/attribute accesses on the file:

# auditctl -w /home/dustymabe/.i3/switch.py -p rwxa -k 'switchtest'
# auditctl -l
-w /home/dustymabe/.i3/switch.py -p rwxa -k switchtest

Notice the usage of the -k option to add a key to the rule. This means any events that match the rule will be tagged with this key and can be easily found. Any accesses will be logged and can be viewed later by using ausearch and aureport. After putting the rules in place in another terminal I accessed the file as a normal user:

$ pwd
/home/dustymabe
$ cat  .i3/switch.py
... contents of file ...
$ ls .i3/switch.py
.i3/switch.py

Then I was able to use a combination of ausearch and aureport to easily see who accessed the file and how it was accessed:

# ausearch -k switchtest --raw | aureport --file

File Report
===============================================
# date time file syscall success exe auid event
===============================================
1. 01/26/15 22:59:26 .i3/switch.py 2 yes /usr/bin/cat 1000 1299
2. 01/26/15 23:00:19 .i3/switch.py 191 no /usr/bin/ls 1000 1300

Awesome.. So with auditing working now all I have to do is press the key sequence to see if my script is getting called?? Turns out it was being called:

# ausearch -k switchtest --raw | aureport --file

File Report
===============================================
# date time file syscall success exe auid event
===============================================
1. 01/26/15 22:59:26 .i3/switch.py 2 yes /usr/bin/cat 1000 1299
2. 01/26/15 23:00:19 .i3/switch.py 191 no /usr/bin/ls 1000 1300
10. 01/26/15 23:38:15 /home/dustymabe/.i3/switch.py 59 yes /usr/bin/python2.7 1000 1326
11. 01/26/15 23:38:15 /home/dustymabe/.i3/switch.py 89 no /usr/bin/python2.7 1000 1327
12. 01/26/15 23:38:15 /home/dustymabe/.i3/switch.py 2 yes /usr/bin/python2.7 1000 1328
13. 01/26/15 23:38:15 /home/dustymabe/.i3/switch.py 2 yes /usr/bin/python2.7 1000 1329
14. 01/26/15 23:38:15 /home/dustymabe/.i3/switch.py 2 yes /usr/bin/python2.7 1000 1330
15. 01/26/15 23:38:15 /home/dustymabe/.i3/switch.py 2 yes /usr/bin/python2.7 1000 1331

So that enabled me to concentrate on my script and find the bug that was lurking within :)

Have fun auditing!
Dusty

Fedora 21 now available on Digital Ocean

cross posted from this fedora magazine post

It's raining Droplets! Fedora 21 has landed in Digital Ocean's cloud hosting. Fedora 21 offers a fantastic cloud image for developers, and it's now easy for Digital Ocean users to spin it up and get started! Here are a couple of tips:

  • Like with other Digital Ocean images, you will log in with your ssh key as root rather than the typical fedora user that you may be familiar with when logging in to a Fedora cloud image.
  • This is the first time Digital Ocean has SELinux enabled by default (yay for security). If you want or need to you can still easily switch back to permissive mode; Red Hat's Dan Walsh may have a "shame on you" or two for you though.
  • Fedora 21 should be available in all the newest datacenters in each region, but some legacy datacenters aren't supported. If you have a problem you think is Fedora specific then drop us an email at , ping us in #fedora-cloud on freenode, or visit the Fedora cloud trac to see if it is already being worked on.

Happy Developing!
Dusty

PS If anyone wants a $10 credit for creating a new account you can use my referral link

qemu-img Backing Files: A Poor Man’s Snapshot/Rollback

I often like to formulate detailed steps when trying to reproduce a bug or a working setup. VMs are great for this because they can be manipulated easily. To manipulate their disk images I use qemu-img to create new disk images that use other disk images as a backing store. This is what I like to call a "poor man's" way to do snapshots because the snapshotting process is a bit manual, but that is also why I like it; I don't touch the original disk image at all so I have full confidence I haven't compromised it.

NOTE: I use QEMU/KVM/Libvirt so those are the tools used in this example:

Taking A Snapshot

In order to take a snapshot you should first shutdown the VM and then simply create a new disk image that uses the original disk image as a backing store:

$ sudo virsh shutdown F21server
Domain F21server is being shutdown
$ sudo qemu-img create -f qcow2 -b /guests/F21server.img /guests/F21server.qcow2.snap
Formatting '/guests/F21server.qcow2.snap', fmt=qcow2 size=21474836480 backing_file='/guests/F21server.img' encryption=off cluster_size=65536 lazy_refcounts=off

This new disk image is a COW snapshot of the original image, which means any writes will go into the new image but any reads of non-modified blocks will be read from the original image. A benefit of this is that the size of the new file will start off at 0 and increase only as modifications are made.

To get the virtual machine to pick up and start using the new COW disk image we will need to modify the libvirt XML to point it at the new file:

$ sudo virt-xml F21server --edit target=vda --disk driver_type=qcow2,path=/guests/F21server.qcow2.snap --print-diff
--- Original XML
+++ Altered XML
@@ -27,8 +27,8 @@
   <devices>
     <emulator>/usr/bin/qemu-kvm</emulator>
     <disk type="file" device="disk">
-      <driver name="qemu" type="raw"/>
-      <source file="/guests/F21server.img"/>
+      <driver name="qemu" type="qcow2"/>
+      <source file="/guests/F21server.qcow2.snap"/>
       <target dev="vda" bus="virtio"/>
       <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/>
     </disk>
$ 
$ sudo virt-xml F21server --edit target=vda --disk driver_type=qcow2,path=/guests/F21server.qcow2.snap
Domain 'F21server' defined successfully.

You can now start your VM and make changes as you wish. Be destructive if you like; the original disk image hasn't been touched.

After making a few changes I had around 15M of differences between the original image and the snapshot:

$ du -sh /guests/F21server.img 
21G     /guests/F21server.img
$ du -sh /guests/F21server.qcow2.snap 
15M     /guests/F21server.qcow2.snap

Going Back

To go back to the point you started you must first delete the file that you created (/guests/F21server.qcow2.snap) and then you have two options:

  • Again create a disk image using the origin as a backing file.
  • Go back to using the original image.

If you want to continue testing and going back to your starting point then you will want to delete and recreate the COW snapshot disk image:

$ sudo rm /guests/F21server.qcow2.snap 
$ sudo qemu-img create -f qcow2 -b /guests/F21server.img /guests/F21server.qcow2.snap
Formatting '/guests/F21server.qcow2.snap', fmt=qcow2 size=21474836480 backing_file='/guests/F21server.img' encryption=off cluster_size=65536 lazy_refcounts=off 

If you want to go back to your original setup then we'll also need to change back the xml to what it was before:

$ sudo rm /guests/F21server.qcow2.snap 
$ sudo virt-xml F21server --edit target=vda --disk driver_type=raw,path=/guests/F21server.img
Domain 'F21server' defined successfully.

Committing Changes

If you happen to decide that the changes you have made are some that you want to carry forward then you can commit the changes in the COW disk image into the backing disk image. In the case below I have 15M worth of changes that get committed back into the original image. I then edit the xml accordingly and can start the guest with all the changes baked back into the original disk image:

$ sudo qemu-img info /guests/F21server.qcow2.snap
image: /guests/F21server.qcow2.snap
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 15M
cluster_size: 65536
backing file: /guests/F21server.img
$ sudo qemu-img commit /guests/F21server.qcow2.snap
Image committed.
$ sudo rm /guests/F21server.qcow2.snap
$ sudo virt-xml F21server --edit target=vda --disk driver_type=raw,path=/guests/F21server.img
Domain 'F21server' defined successfully.

Fin

This backing file approach is useful because it's much more convenient than making multiple copies of huge disk image files, but it can be used for much more than just snapshotting/reverting changes. It can also be used to start 100 virtual machines from a common backing image, thus saving space...etc.. Go ahead and try it!

Happy Snapshotting!
Dusty

F21 Atomic Test Day && Test steps for Atomic Host

Test Day on Thursday 11/20

The F21 test day for atomic is this Thursday, November 20th. If anyone can participate please do drop into #atomic on freenode as it will be great to have more people involved in helping build/test this new technology.

In anticipation of the test day I have put together some test notes for other people to follow in hopes that it will help smooth things along.

Booting with cloud-init

First step is to start an atomic host using any method/cloud provider you like. For me I decided to use openstack since I have Juno running on F21 here in my apartment. I used this user-data for the atomic host:

#cloud-config password: passw0rd chpasswd: { expire: False } ssh_pwauth: True runcmd: - [ sh, -c, 'echo -e "ROOT_SIZE=4GnDATA_SIZE=10G" > /etc/sysconfig/docker-storage-setup']

Note that the build of atomic I used for this testing resides here

Verifying docker-storage-setup

docker-storage-setup is a service that can be used to configure the storage configuration for docker in different ways on instance bringup. Notice in the user-data above that I decided to set config variables for docker-storage-setup. They basically mean that I want to resize my atomicos/root LV to 4G and I want to create an atomicos/docker-data LV and make it 10G in size.

To verify the storage was set up successfully, log in (as the fedora user) and become root (usind sudo su -). Now you can check if docker-storage-setup worked by checking the logs as well as looking at the output from lsblk:

# journalctl -o cat --unit docker-storage-setup.service CHANGED: partition=2 start=411648 old: size=12171264 end=12582912 new: size=41531232,end=41942880 Physical volume "/dev/vda2" changed 1 physical volume(s) resized / 0 physical volume(s) not resized Size of logical volume atomicos/root changed from 1.95 GiB (500 extents) to 4.00 GiB (1024 extents). Logical volume root successfully resized Rounding up size to full physical extent 24.00 MiB Logical volume "docker-meta" created Logical volume "docker-data" created # # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 252:0 0 20G 0 disk ├─vda1 252:1 0 200M 0 part /boot └─vda2 252:2 0 19.8G 0 part ├─atomicos-root 253:0 0 4G 0 lvm /sysroot ├─atomicos-docker--meta 253:1 0 24M 0 lvm └─atomicos-docker--data 253:2 0 10G 0 lvm

Verifying Docker Lifecycle

To verify Docker runs fine on the atomic host we will perform a simple run of the busybox docker image. This will contact the docker hub, pull down the image, and run /bin/true:

# docker run -it --rm busybox true && echo "PASS" || echo "FAIL" Unable to find image 'busybox' locally Pulling repository busybox e72ac664f4f0: Download complete 511136ea3c5a: Download complete df7546f9f060: Download complete e433a6c5b276: Download complete PASS

After the Docker daemon has started the LVs that were created by docker-storage-setup will be used by device mapper as shown in the lsblk output below:

# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 252:0 0 20G 0 disk ├─vda1 252:1 0 200M 0 part /boot └─vda2 252:2 0 19.8G 0 part ├─atomicos-root 253:0 0 4G 0 lvm /sysroot ├─atomicos-docker--meta 253:1 0 24M 0 lvm │ └─docker-253:0-6298462-pool 253:3 0 10G 0 dm │ └─docker-253:0-6298462-base 253:4 0 10G 0 dm └─atomicos-docker--data 253:2 0 10G 0 lvm └─docker-253:0-6298462-pool 253:3 0 10G 0 dm └─docker-253:0-6298462-base 253:4 0 10G 0 dm

Atomic Host: Upgrade

Now on to an atomic upgrade. First let's check what commit we are currently at and store a file in /etc/file1 to save it for us:

# rpm-ostree status TIMESTAMP (UTC) ID OSNAME REFSPEC * 2014-11-12 22:28:04 1877f1fa64 fedora-atomic-host fedora-atomic:fedora-atomic/f21/x86_64/docker-host # # ostree admin status * fedora-atomic-host 1877f1fa64be8bec8adcd43de6bd4b5c39849ec7842c07a6d4c2c2033651cd84.0 origin refspec: fedora-atomic:fedora-atomic/f21/x86_64/docker-host # # cat /ostree/repo/refs/heads/ostree/0/1/0 1877f1fa64be8bec8adcd43de6bd4b5c39849ec7842c07a6d4c2c2033651cd84 # # cat /ostree/repo/refs/heads/ostree/0/1/0 > /etc/file1

Now run an upgrade to the latest atomic compose:

# rpm-ostree upgrade Updating from: fedora-atomic:fedora-atomic/f21/x86_64/docker-host 14 metadata, 19 content objects fetched; 33027 KiB transferred in 16 seconds Copying /etc changes: 26 modified, 4 removed, 39 added Transaction complete; bootconfig swap: yes deployment count change: 1) Updates prepared for next boot; run "systemctl reboot" to start a reboot

And do a bit of poking around right before we reboot:

# rpm-ostree status TIMESTAMP (UTC) ID OSNAME REFSPEC 2014-11-13 10:52:06 18e02c4166 fedora-atomic-host fedora-atomic:fedora-atomic/f21/x86_64/docker-host * 2014-11-12 22:28:04 1877f1fa64 fedora-atomic-host fedora-atomic:fedora-atomic/f21/x86_64/docker-host # # ostree admin status fedora-atomic-host 18e02c41666ef5f426bc43d01c4ce1b7ffc0611e993876cf332600e2ad8aa7c0.0 origin refspec: fedora-atomic:fedora-atomic/f21/x86_64/docker-host * fedora-atomic-host 1877f1fa64be8bec8adcd43de6bd4b5c39849ec7842c07a6d4c2c2033651cd84.0 origin refspec: fedora-atomic:fedora-atomic/f21/x86_64/docker-host # # reboot

Note that the * in the above output indicates which tree is currently booted.

After reboot now the new tree should be booted. Let's check things out and make /etc/file2 with our new commit hash in it:

# rpm-ostree status TIMESTAMP (UTC) ID OSNAME REFSPEC * 2014-11-13 10:52:06 18e02c4166 fedora-atomic-host fedora-atomic:fedora-atomic/f21/x86_64/docker-host 2014-11-12 22:28:04 1877f1fa64 fedora-atomic-host fedora-atomic:fedora-atomic/f21/x86_64/docker-host # # ostree admin status * fedora-atomic-host 18e02c41666ef5f426bc43d01c4ce1b7ffc0611e993876cf332600e2ad8aa7c0.0 origin refspec: fedora-atomic:fedora-atomic/f21/x86_64/docker-host fedora-atomic-host 1877f1fa64be8bec8adcd43de6bd4b5c39849ec7842c07a6d4c2c2033651cd84.0 origin refspec: fedora-atomic:fedora-atomic/f21/x86_64/docker-host # # cat /ostree/repo/refs/heads/ostree/1/1/0 18e02c41666ef5f426bc43d01c4ce1b7ffc0611e993876cf332600e2ad8aa7c0 # # cat /ostree/repo/refs/heads/ostree/1/1/0 > /etc/file2

As one final item let's boot up a docker container to make sure things still work there:

# docker run -it --rm busybox true && echo "PASS" || echo "FAIL" PASS

Atomic Host: Rollback

Atomic host provides the ability to revert to the previous working tree if things go awry with the new tree. Lets revert our upgrade now and make sure things still work:

# rpm-ostree rollback Moving '1877f1fa64be8bec8adcd43de6bd4b5c39849ec7842c07a6d4c2c2033651cd84.0' to be first deployment Transaction complete; bootconfig swap: yes deployment count change: 0) Sucessfully reset deployment order; run "systemctl reboot" to start a reboot # # rpm-ostree status TIMESTAMP (UTC) ID OSNAME REFSPEC 2014-11-12 22:28:04 1877f1fa64 fedora-atomic-host fedora-atomic:fedora-atomic/f21/x86_64/docker-host * 2014-11-13 10:52:06 18e02c4166 fedora-atomic-host fedora-atomic:fedora-atomic/f21/x86_64/docker-host # # reboot

After reboot:

# rpm-ostree status TIMESTAMP (UTC) ID OSNAME REFSPEC * 2014-11-12 22:28:04 1877f1fa64 fedora-atomic-host fedora-atomic:fedora-atomic/f21/x86_64/docker-host 2014-11-13 10:52:06 18e02c4166 fedora-atomic-host fedora-atomic:fedora-atomic/f21/x86_64/docker-host # # cat /etc/file1 1877f1fa64be8bec8adcd43de6bd4b5c39849ec7842c07a6d4c2c2033651cd84 # cat /etc/file2 cat: /etc/file2: No such file or directory

Notice that /etc/file2 did not exist until after the upgrade so it did not persist during the rollback.

And the final item on the list is to make sure Docker still works:

# docker run -it --rm busybox true && echo "PASS" || echo "FAIL" PASS

Anddd Boom.. You have just put atomic through some paces.