cross posted with this Project Atomic blog post and this Fedora Magazine post
Introduction
In July we put out the first and second releases of Fedora 26 Atomic Host. In this blog post we’ll cover updating an existing Fedora 25 Atomic Host system to Fedora 26. We’ll cover preparing the system for upgrade and performing the upgrade.
NOTE: If you really don’t want to upgrade to Fedora 26 see the later section: Appendix B: Fedora 25 Atomic Host Life Support.
Preparing for Upgrade
Before we update to Fedora 26 Atomic Host we should check to see that we have at least a few GiB of free space in our root filesystem. The update to Fedora 26 can cause your system to retrieve more than 1GiB of new content (not shared with Fedora 25) and thus we’ll need to make sure we have plenty of free space.
NOTE: Upstream OSTree has implemented some filesystem checks to make sure that upgrades will stop themselves before filling up the filesystem and possibly corrupting your system.
The system we are upgrading today is a Vagrant box. Let’s see how much free space we have:
[vagrant@host ~]$ sudo df -kh /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/atomicos-root 3.0G 1.4G 1.6G 47% /
Only 1.6G
free means we probably need to expand our root filesystem
to make sure we don’t run out of space. Let’s check to see if we have
any free space:
[vagrant@host ~]$ sudo vgs
VG #PV #LV #SN Attr VSize VFree
atomicos 1 2 0 wz--n- 40.70g 22.60g
[vagrant@host ~]$ sudo lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
docker-pool atomicos twi-a-t--- 15.09g 0.13 0.10
root atomicos -wi-ao---- 2.93g
We can see that we have 22.60g
free and that our atomicos/root
logical volume is 2.93g
in size. We’ll go ahead and increase the
size of the root volume group by 3 GiB.
[vagrant@host ~]$ sudo lvresize --size=+3g --resizefs atomicos/root
Size of logical volume atomicos/root changed from 2.93 GiB (750 extents) to 5.93 GiB (1518 extents).
Logical volume atomicos/root successfully resized.
meta-data=/dev/mapper/atomicos-root isize=512 agcount=4, agsize=192000 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1 spinodes=0 rmapbt=0
= reflink=0
data = bsize=4096 blocks=768000, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 768000 to 1554432
[vagrant@host ~]$ sudo lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
docker-pool atomicos twi-a-t--- 15.09g 0.13 0.10
root atomicos -wi-ao---- 5.93g
As part of that command we also resized the filesystem all in one shot. We can see that by checking again the filesystem usage.
[vagrant@host ~]$ sudo df -kh /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/atomicos-root 6.0G 1.4G 4.6G 24% /
Upgrading
Now we should be ready for the upgrade. If you are hosting any services on your instance you may want to prepare for them to have some downtime.
NOTE: If you are running Kubernetes you should check out the later section on Kubernetes: Appendix A: Upgrading Systems with Kubernetes.
NOTE: If you are running OpenShift Origin (i.e. via being set up by the openshift-ansible installer) the upgrade should not need any preparation.
Currently we are on Fedora 25 Atomic Host using the
fedora-atomic/25/x86_64/docker-host
ref.
[vagrant@host ~]$ rpm-ostree status
State: idle
Deployments:
● fedora-atomic:fedora-atomic/25/x86_64/docker-host
Version: 25.154 (2017-07-04 01:38:10)
Commit: ce555fa89da934e6eef23764fb40e8333234b8b60b6f688222247c958e5ebd5b
In order to do the upgrade we need to add the location of
the Fedora 26 repository as a new remote (similar to a
git remote) for ostree
to know about:
[vagrant@host ~]$ sudo ostree remote add --set=gpgkeypath=/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-26-primary fedora-atomic-26 https://kojipkgs.fedoraproject.org/atomic/26
You can see from the command that we are adding a new remote known as
fedora-atomic-26
with a remote url of https://kojipkgs.fedoraproject.org/atomic/26
.
We are also setting the gpgkeypath
variable in the configuration for
the remote. This tells OSTree that we want commit signatures to be
verified when we download from a remote. This is something new that was
enabled for Fedora 26 Atomic Host.
Now that we have our fedora-atomic-26
remote we can do the upgrade!
[vagrant@host ~]$ sudo rpm-ostree rebase fedora-atomic-26:fedora/26/x86_64/atomic-host
Receiving metadata objects: 0/(estimating) -/s 0 bytes
Signature made Sun 23 Jul 2017 03:13:09 AM UTC using RSA key ID 812A6B4B64DAB85D
Good signature from "Fedora 26 Primary <fedora-26-primary@fedoraproject.org>"
Receiving delta parts: 0/27 5.3 MB/s 26.7 MB/355.4 MB
Signature made Sun 23 Jul 2017 03:13:09 AM UTC using RSA key ID 812A6B4B64DAB85D
Good signature from "Fedora 26 Primary <fedora-26-primary@fedoraproject.org>"
27 delta parts, 9 loose fetched; 347079 KiB transferred in 105 seconds
Copying /etc changes: 22 modified, 0 removed, 58 added
Transaction complete; bootconfig swap: yes deployment count change: 1
Upgraded:
GeoIP 1.6.11-1.fc25 -> 1.6.11-1.fc26
GeoIP-GeoLite-data 2017.04-1.fc25 -> 2017.06-1.fc26
NetworkManager 1:1.4.4-5.fc25 -> 1:1.8.2-1.fc26
...
...
setools-python-4.1.0-3.fc26.x86_64
setools-python3-4.1.0-3.fc26.x86_64
Run "systemctl reboot" to start a reboot
[vagrant@host ~]$ sudo reboot
Connection to 192.168.121.217 closed by remote host.
Connection to 192.168.121.217 closed.
After reboot we can log in and see the status:
$ vagrant ssh
[vagrant@host ~]$ rpm-ostree status
State: idle
Deployments:
● fedora-atomic-26:fedora/26/x86_64/atomic-host
Version: 26.91 (2017-07-23 03:12:08)
Commit: 0715ce81064c30d34ed52ef811a3ad5e5d6a34da980bf35b19312489b32d9b83
GPGSignature: 1 signature
Signature made Sun 23 Jul 2017 03:13:09 AM UTC using RSA key ID 812A6B4B64DAB85D
Good signature from "Fedora 26 Primary <fedora-26-primary@fedoraproject.org>"
fedora-atomic:fedora-atomic/25/x86_64/docker-host
Version: 25.154 (2017-07-04 01:38:10)
Commit: ce555fa89da934e6eef23764fb40e8333234b8b60b6f688222247c958e5ebd5b
[vagrant@host ~]$ cat /etc/fedora-release
Fedora release 26 (Twenty Six)
We are now on Fedora 26 Atomic Host. Now is a good time to check your
services (most likely running in containers) to see if they are still
working. If not, then you always have the rollback command: sudo rpm-ostree rollback
.
NOTE: Over time you can see updated commands for upgrading Atomic Host between releases by visiting this wiki page.
Appendix A: Upgrading Systems with Kubernetes
Fedora 25 Atomic Host ships with Kubernetes v1.5.3, and Fedora 26 Atomic Host ships with Kubernetes v1.6.7. Before upgrading systems participating in an existing Kubernetes cluster from 25 to 26, there are a few configuration changes to make.
Node Servers
In Kubernetes 1.6, the --config
argument is no longer valid. If
your KUBELET_ARGS
in /etc/kubernetes/kubelet
point to the manifests
directory using the --config
argument, then you need to change
the argument name to --pod-manifest-path
. Also in KUBELET_ARGS
, you
need to add the argument --cgroup-driver=systemd
.
For example, if your /etc/kubernetes/kubelet
file starts with the
following:
KUBELET_ARGS="--kubeconfig=/etc/kubernetes/kubelet.kubeconfig --config=/etc/kubernetes/manifests --cluster-dns=10.254.0.10 --cluster-domain=cluster.local"
Then it should be changed to be:
KUBELET_ARGS="--kubeconfig=/etc/kubernetes/kubelet.kubeconfig --pod-manifest-path=/etc/kubernetes/manifests --cluster-dns=10.254.0.10 --cluster-domain=cluster.local --cgroup-driver=systemd"
Master Servers
Staying With etcd2
From Kubernetes 1.5 to 1.6 upstream
shifted
from using version 2 of the etcd API to version 3. The
Kubernetes documentation
instructs users to add two arguments to the KUBE_API_ARGS
variable
in the /etc/kubernetes/apiserver
file:
--storage-backend=etcd2 --storage-media-type=application/json
This will ensure that any pods, services or other objects stored in etcd will continue to be found by Kubernetes once you’ve completed your upgrade.
Moving To etcd3
If you later wish to migrate your etcd data to the v3 API, stop your etcd and kube-apiserver services and, run the following command to migrate to etcd3:
NOTE: The following command assumes your data is stored in
/var/lib/etcd
.
# ETCDCTL_API=3 etcdctl --endpoints https://YOUR-ETCD-IP:2379 migrate --data-dir=/var/lib/etcd
Then remove the --storage-backend=etcd2
and --storage-media-type=application/json
arguments from the /etc/kubernetes/apiserver
file and restart the etcd
and kube-apiserver services.
Appendix B: Fedora 25 Atomic Host Life Support
We have decided
to keep updating the fedora-atomic/25/x86_64/docker-host
ref every day when Bodhi runs within Fedora. A new update will
get created every day. However, we recommend you upgrade to Fedora 26,
because we are focusing future testing and development efforts on on
Fedora 26 Atomic Host and thus the Fedora 25 OSTrees don’t get
tested.
Conclusion
The transition to Fedora 26 Atomic Host should be a smooth process. If you have issues or want to be involved in the future direction of Atomic Host please join us in IRC (#atomic on freenode) or on the atomic-devel mailing list.