cross posted with this Project Atomic blog post
In this post we’ll quickly list some known issues and then talk about updating an existing Fedora 28 Atomic Host system to Fedora 29. We’ll cover preparing the system for upgrade and performing the upgrade.
NOTE: If you really don’t want to upgrade to Fedora 29 see the later section: Appendix B: Fedora 28 Atomic Host Life Support.
- Device timeout when installing from ISO with swap space.
We found an issue where instances installed via the ISO image via anaconda were having long boot times because of device timeouts. We traced this down to a bug in dracut and this will be fixed in the next release.
- Incompatibilities with OpenShift Origin >= 3.11 and Kubernetes >= 1.10.4
In newer versions of
systemd an API was removed that was needed by
older versions of OpenShift/Kubernetes. It is recommended that you
upgrade to Origin 3.11 or Kubernetes > 1.10.4 before moving to Fedora
29 Atomic Host. If you must use an older version of OpenShift/Kubernetes
with Fedora 29 Atomic Host please see the later section:
Appendix A: Using older versions of Kubernetes/OpenShift on Fedora 29 Atomic Host.
Discussion of this issue can be found in the pagure ticket.
Preparing for Upgrade
Before we update to Fedora 29 Atomic Host we should check to see that we have at least a few GiB of free space in our root filesystem. The update to Fedora 29 can cause your system to retrieve more than 1 GiB of new content (not shared with Fedora 28) and thus we’ll need to make sure we have plenty of free space.
NOTE: Luckily if you skip this step OSTree will perform filesystem checks to make sure that upgrade will error gracefully before filling up the filesystem.
Now we should be ready for the upgrade. If you are hosting any services on your instance you may want to prepare for them to have some downtime.
Currently we are on Fedora 28 Atomic Host using the
[vagrant@host ~]$ rpm-ostree status State: idle AutomaticUpdates: disabled Deployments: ● ostree://fedora-atomic:fedora/28/x86_64/atomic-host Version: 28.20181007.0 (2018-10-07 20:43:44) Commit: 8df48fa2e70ad1952153ae00edbba08ed18b53c3d4095a22985d1085f5203ac6 GPGSignature: Valid signature by 128CF232A9371991C8A65695E08E7E629DB62FB1
In order to do the upgrade we need to add the location of
the Fedora 28 repository as a new remote (similar to a
git remote) for
ostree to know about:
[vagrant@host ~]$ sudo ostree remote add --set=gpgkeypath=/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-29-primary fedora-atomic-29 https://dl.fedoraproject.org/atomic/repo/
You can see from the command that we are adding a new remote known as
fedora-atomic-29 with a remote url of
We are also setting the
gpgkeypath variable in the configuration for
the remote. This tells OSTree that we want commit signatures to be
verified when we download from a remote.
Now that we have our
fedora-atomic-29 remote we can do the upgrade!
[vagrant@host ~]$ sudo rpm-ostree rebase fedora-atomic-29:fedora/29/x86_64/atomic-host 26 delta parts, 9 loose fetched; 264197 KiB transferred in 44 seconds Upgraded: NetworkManager 1:1.10.12-1.fc28 -> 1:1.12.4-1.fc29 NetworkManager-libnm 1:1.10.12-1.fc28 -> 1:1.12.4-1.fc29 NetworkManager-team 1:1.10.12-1.fc28 -> 1:1.12.4-1.fc29 acl 2.2.53-1.fc28 -> 2.2.53-2.fc29 atomic 1.22.1-2.fc28 -> 1.22.1-27.gitb507039.fc29 atomic-devmode 0.3.7-4.fc28 -> 0.3.7-6.fc29 atomic-registries 1.22.1-2.fc28 -> 1.22.1-27.gitb507039.fc29 attr 2.4.48-3.fc28 -> 2.4.48-3.fc29 ... python-unversioned-command-2.7.15-10.fc29.noarch python3-pyyaml-4.2-0.1.b4.fc29.x86_64 Run "systemctl reboot" to start a reboot [vagrant@host ~]$ sudo systemctl reboot [vagrant@host ~]$ Connection to 192.168.121.95 closed by remote host. Connection to 192.168.121.95 closed.
After reboot we can log in and see the status:
$ vagrant ssh Last login: Wed Oct 31 21:14:44 2018 from 192.168.121.1 [vagrant@host ~]$ rpm-ostree status State: idle AutomaticUpdates: disabled Deployments: ● ostree://fedora-atomic-29:fedora/29/x86_64/atomic-host Version: 29.20181025.1 (2018-10-25 14:46:54) Commit: 4a999b4b303b47468ff1464051a14fd075d2e7b8bb647584b7cc80fed48cf27b GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4 ostree://fedora-atomic:fedora/28/x86_64/atomic-host Version: 28.20181007.0 (2018-10-07 20:43:44) Commit: 8df48fa2e70ad1952153ae00edbba08ed18b53c3d4095a22985d1085f5203ac6 GPGSignature: Valid signature by 128CF232A9371991C8A65695E08E7E629DB62FB1 [vagrant@host ~]$ [vagrant@host ~]$ cat /etc/os-release | head -n 2 NAME=Fedora VERSION="29.20181025.1 (Atomic Host)"
We are now on Fedora 29 Atomic Host. Now is a good time to check your
services (most likely running in containers) to see if they are still
working. If not, then you always have the rollback command:
NOTE: Over time you can see updated commands for upgrading Atomic Host between releases by visiting this wiki page.
Appendix A: Using older versions of Kubernetes/OpenShift on Fedora 29 Atomic Host
Due to an issue that arises from the combination of the version of systemd that ships with Fedora 29 Atomic Host, and the version of runc that’s bundled with Kubernetes and with OpenShift Origin, users of certain Kubernetes and OpenShift Origin releases must take steps to keep their Kubernetes or Origin clusters running following an upgrade to 29.
Users running Kubernetes releases older than v1.10.4 or OpenShift Origin releases older than v3.11 would do best to upgrade their Kubernetes or Origin clusters to a more recent version before undertaking an upgrade to Fedora 29 Atomic Host.
For users who prefer to upgrade to 29 without changing their cluster
version, it’s possible to work around the issue by changing the
cgroupdriver for both docker and for either the kubelet or the
orgin-node component from its default value of
For docker (applies to both Kubernetes and OpenShift Origin clusters):
# cp /usr/lib/systemd/system/docker.service /etc/systemd/system/ # sed -i 's/cgroupdriver=systemd/cgroupdriver=cgroupfs/' /etc/systemd/system/docker.service
For kubelet (applies to Kubeadm-based Kubernetes clusters):
# sed -i 's/cgroup-driver=systemd/cgroup-driver=cgroupfs/' /etc/systemd/system/kubelet.service.d/kubeadm.conf
For origin-node (applies to OpenShift Origin clusters):
# vi /etc/origin/node/node-config.yaml
Add the following element to the
cgroup-driver: - "cgroupfs"
After making these changes, rebase to 29 as directed above. After reboot
the nodes should return in
Ready state as expected.
Appendix B: Fedora 28 Atomic Host Life Support
As with Fedora 27 we will keep updating the Fedora 28 Atomic Host tree but we won’t be performing regular testing on that tree and there will be no more formal releases for Fedora 28. Please move to Fedora 29 to take advantage of regular testing and formal releases every two weeks.