Red Hat Summit was a blast this year. I participated in several Hands On Labs to help the community learn about the new tools that are available in the ecosystem. For one of the labs I wrote up a section on Atomic Host, but more specifically on rpm-ostree. I have copied a portion of the lab here as well as added example text to the code blocks.
Lab Intro
Atomic Host is a minimalistic operating system that is designed to contain a very small subset of tools that are needed for running container based applications. A few of it's features are shown below:
- It is Lightweight
- a small base means less potential issues.
- Provides Atomic Upgrades and Rollbacks
- upgrades/rollbacks are staged and take effect on reboot
- Static and Dynamic
- software/binaries in /usr and other similar directories are read-only
- this guarantees no changes have been made to the software
- configuration and temporary directories are read/write
- you can still make important configuration changes and have them propagate forward
- software/binaries in /usr and other similar directories are read-only
We will explore some of these features as we illustrate a bit of the lifecycle of managing a RHEL Atomic Host.
Hello rpm-ostree World
In an rpm-ostree world you can't install new software on the system or even touch most of the software that exists. Go ahead and try:
-bash-4.2# echo 'Crazy Talk' > /usr/bin/docker -bash: /usr/bin/docker: Read-only file system
What we can do is configure the existing software on the system using the provided mechanisms for configuration. We can illustrate this by writing to motd and then logging in to see the message:
-bash-4.2# echo 'Lab 1 is fun' > /etc/motd -bash-4.2# ssh root@localhost Last login: Fri Jun 5 02:26:59 2015 from localhost Lab 1 is fun -bash-4.2# exit logout Connection to localhost closed.
Even though we can't install new software, your Atomic Host operating system isn't just a black box. The rpm command is there and we can run queries just the same as if we were on a traditional system. This is quite useful because we can use the tools we are familiar with to investigate the system. Try out a few rpm queries on the Atomic Host:
-bash-4.2# rpm -q kernel kernel-3.10.0-229.4.2.el7.x86_64 -bash-4.2# rpm -qf /usr/bin/vi vim-minimal-7.4.160-1.el7.x86_64 -bash-4.2# rpm -q --changelog util-linux | wc -l 1832
Another nice thing about atomic, or rather the underlying ostree software is that it is like git for your OS. At any point in time you can see what has changed between what was delivered in the tree vs. what is on the system. That means for those few directories that are read/write, you can easily view what changes have been made to them.
Let's take a look at the existing differences between what we have and what was delivered in the tree:
-bash-4.2# ostree admin config-diff | head -n 5 M adjtime M motd M group M hosts M gshadow
You can see right in the middle the the motd file we just modified.
As a final step before we do an upgrade let's run a container and verify all is working:
-bash-4.2# docker run -d -p 80:80 --name=test repo.atomic.lab:5000/apache e18a5f7d54c8dbe0d352e2c2854af16d27f166d11b95bc37a3b4267cfcd39cd6 -bash-4.2# curl http://localhost Apache -bash-4.2# docker rm -f test test
Performing an Upgrade
Ok, now that we have took a little tour, let's actually perform an upgrade in which we move from one version of the tree to a newer version. First, let's check the current status of the system:
-bash-4.2# atomic host status TIMESTAMP (UTC) VERSION ID OSNAME REFSPEC * 2015-05-30 04:10:40 d306dcf255 rhel-atomic-host lab:labtree 2015-05-07 19:00:48 7.1.2 203dd666d3 rhel-atomic-host rhel-atomic-host-ostree:rhel-...
Note that the * indicates which tree is currently booted. The ID is a short commit ID for that commit in the tree. The REFSPEC for the latest tree specifies the remote we are using (lab) and the ref that we are tracking (labtree). Quite a lot of information!
A fun fact is that the atomic host command is just a frontend for the rpm-ostree utility. It has some of the functionality of the rpm-ostree utility that is suitable for most daily use. Let's use rpm-ostree now to check the status:
-bash-4.2# rpm-ostree status TIMESTAMP (UTC) VERSION ID OSNAME REFSPEC * 2015-05-30 04:10:40 d306dcf255 rhel-atomic-host lab:labtree 2015-05-07 19:00:48 7.1.2 203dd666d3 rhel-atomic-host rhel-atomic-host-ostree:rhel-...
The next step is to actually move to a new tree. For the purposes of this lab, and to illustrate Atomic's usefulness, we are actually going to upgrade to a tree that has some bad software in it. If we were to run an atomic host upgrade command then it would actually take us to the newest commit in the repo. In this case we want to go to an intermediate commit (a bad one) so we are going to run a special command to get us there:
-bash-4.2# rpm-ostree rebase lab:badtree 26 metadata, 37 content objects fetched; 101802 KiB transferred in 7 seconds Copying /etc changes: 26 modified, 8 removed, 70 added Transaction complete; bootconfig swap: yes deployment count change: 0 Freed objects: 180.1 MB Deleting ref 'lab:labtree' Changed: etcd-2.0.11-2.el7.x86_64 kubernetes-0.17.1-1.el7.x86_64 Removed: setools-console-3.3.7-46.el7.x86_64
What we did there was rebase to another ref (badtree), but we kept with the same remote (lab).
So we have rebased to a new tree but we aren't yet using that tree. During upgrade the new environment is staged for the next boot, but not yet being used. This allows the upgrade to be atomic. Before we reboot we can check the status. You will see the new tree as well as the old tree listed. The * still should be next to the old tree since that is the tree that is currently booted and running:
-bash-4.2# atomic host status TIMESTAMP (UTC) ID OSNAME REFSPEC 2015-05-30 04:39:22 146b72d9d7 rhel-atomic-host lab:badtree * 2015-05-30 04:10:40 d306dcf255 rhel-atomic-host lab:labtree
After checking the status reboot the machine in order to boot into the new tree.
Rolling Back
So why would you ever need to roll back? It's a perfect world and nothing ever breaks right? No! Sometimes problems arise and it is always nice to have an undo button to fix it. In the case of Atomic, there is atomic host rollback. Do we need to use it now? Let's see if everything is OK on the system:
-bash-4.2# atomic host status TIMESTAMP (UTC) ID OSNAME REFSPEC * 2015-05-30 04:39:22 146b72d9d7 rhel-atomic-host lab:badtree 2015-05-30 04:10:40 d306dcf255 rhel-atomic-host lab:labtree -bash-4.2# -bash-4.2# docker run -d -p 80:80 --name=test repo.atomic.lab:5000/apache ERROR -bash-4.2# curl http://localhost curl: (7) Failed connect to localhost:80; Connection refused -bash-4.2# systemctl --failed | head -n 3 UNIT LOAD ACTIVE SUB DESCRIPTION docker.service loaded failed failed Docker Application Container Engine
Did anything fail? Of course it did. So let's press the eject button and get ourselves back to safety:
-bash-4.2# atomic host rollback Moving 'd306dcf255b370e5702206d064f2ca2e24d1ebf648924d52a2e00229d5b08365.0' to be first deployment Transaction complete; bootconfig swap: yes deployment count change: 0 Changed: etcd-2.0.9-2.el7.x86_64 kubernetes-0.15.0-0.4.git0ea87e4.el7.x86_64 Added: setools-console-3.3.7-46.el7.x86_64 Sucessfully reset deployment order; run "systemctl reboot" to start a reboot -bash-4.2# reboot
Now, let's check to see if we are back to a good state:
-bash-4.2# atomic host status TIMESTAMP (UTC) ID OSNAME REFSPEC * 2015-05-30 04:10:40 d306dcf255 rhel-atomic-host lab:labtree 2015-05-30 04:39:22 146b72d9d7 rhel-atomic-host lab:badtree -bash-4.2# docker run -d -p 80:80 --name=test repo.atomic.lab:5000/apache a28a5f80bc2d1da9d405199f88951a62a7c4c125484d30fbb6eb2c4c032ef7f3 -bash-4.2# curl http://localhost Apache -bash-4.2# docker rm -f test test
All dandy!
Final Upgrade
So since the badtree has been released the developers fixed the bug and have put out a new tree that is fixed. Now we can upgrade to the newest tree. As part of this upgrade let's explore some of the rpm-ostree features.
First, create a file in /etc/ and show that ostree knows that it has been created and differs from the tree that was delivered:
-bash-4.2# echo "Before Upgrade d306dcf255" > /etc/before-upgrade.txt -bash-4.2# ostree admin config-diff | grep before-upgrade A before-upgrade.txt
Now we can do the upgrade:
-bash-4.2# atomic host upgrade --reboot Updating from: lab:labtree 48 metadata, 54 content objects fetched; 109056 KiB transferred in 9 seconds Copying /etc changes: 26 modified, 8 removed, 74 added Transaction complete; bootconfig swap: yes deployment count change: 0
After the upgrade let's actually run a few commands to see the actual difference is (in terms of rpms) between the two trees:
-bash-4.2# atomic host status TIMESTAMP (UTC) ID OSNAME REFSPEC * 2015-05-30 05:12:55 ec89f90273 rhel-atomic-host lab:labtree 2015-05-30 04:10:40 d306dcf255 rhel-atomic-host lab:labtree -bash-4.2# rpm-ostree db diff -F diff d306dcf255 ec89f90273 ostree diff commit old: d306dcf255 (d306dcf255b370e5702206d064f2ca2e24d1ebf648924d52a2e00229d5b08365) ostree diff commit new: ec89f90273 (ec89f902734e70b4e8fbe5000e87dd944a3c95ffdb04ef92f364e5aaab049813) !atomic-0-0.22.git5b2fa8d.el7.x86_64 =atomic-0-0.26.gitcc9aed4.el7.x86_64 !docker-1.6.0-11.el7.x86_64 =docker-1.6.0-15.el7.x86_64 !docker-python-1.0.0-35.el7.x86_64 =docker-python-1.0.0-39.el7.x86_64 !docker-selinux-1.6.0-11.el7.x86_64 =docker-selinux-1.6.0-15.el7.x86_64 !docker-storage-setup-0.0.4-2.el7.noarch =docker-storage-setup-0.5-2.el7.x86_64 !etcd-2.0.9-2.el7.x86_64 =etcd-2.0.11-2.el7.x86_64 !kubernetes-0.15.0-0.4.git0ea87e4.el7.x86_64 =kubernetes-0.17.1-4.el7.x86_64 +kubernetes-master-0.17.1-4.el7.x86_64 +kubernetes-node-0.17.1-4.el7.x86_64 !python-websocket-client-0.14.1-78.el7.noarch =python-websocket-client-0.14.1-82.el7.noarch -setools-console-3.3.7-46.el7.x86_64
This shows added, removed, changed rpms between the two trees.
Now remember that file we created before the upgrade? Is it still there? Let's check and also create a new file that represents the after upgrade state:
-bash-4.2# cat /etc/before-upgrade.txt Before Upgrade d306dcf255 -bash-4.2# echo "After Upgrade ec89f90273" > /etc/after-upgrade.txt -bash-4.2# cat /etc/after-upgrade.txt After Upgrade ec89f90273
Now which of the files do you think will exist after a rollback? Only you can find out!:
-bash-4.2# rpm-ostree rollback --reboot Moving 'd306dcf255b370e5702206d064f2ca2e24d1ebf648924d52a2e00229d5b08365.0' to be first deployment Transaction complete; bootconfig swap: yes deployment count change: 0
After rollback:
-bash-4.2# atomic host status TIMESTAMP (UTC) ID OSNAME REFSPEC * 2015-05-30 04:10:40 d306dcf255 rhel-atomic-host lab:labtree 2015-05-30 05:12:55 ec89f90273 rhel-atomic-host lab:labtree -bash-4.2# ls -l /etc/*.txt -rw-r--r--. 1 root root 26 Jun 5 03:35 /etc/before-upgrade.txt
Fin!
Now you know quite a bit about upgrading, rolling back, and querying information from your Atomic Host. Have fun exploring!