a more permanent version of this content lives here
Introduction
NOTE: For background on OSTree check out the docs.
When you want to create a new OSTree using rpm-ostree you usually define
a few yum repos, and then a json file that says what rpms you
want to be composed in the tree. You then run an rpm-ostree compose tree command to create the commit in the ostree repo. Once the
ostree commit has been created you can then create installer images
(ISOs) and cloud/VM images (qcow, etc) from that ostree.
How does Fedora do this? It’s a bit complicated, but I’ll try to cover the bases.
First I’ll give you a little high level view of the way things are built before major release, all using Pungi. I’ll then try to dig into The LEGOs that are used to build the artifacts we care about. Finally, I’ll try to explain how things are different after a major release is out and how we end up getting updated rpms and eventually two week releases out the door.
Before Fedora Major Release: Building Everything Using Pungi
Prior to any major release of Fedora the release engineering team builds the whole world every night. This is done using a tool called Pungi. A script is called to kick off the nightly run via a cron job. The Pungi config that is used is the fedora.conf file from the pungi-fedora git repo.
For Fedora 26 you can see where the ostree for Atomic Host is defined
in the Pungi config, here,
in the ostree section. During a compose you can
see that the ostree will get placed into "ostree_repo": "/mnt/koji/compose/atomic/26/",.
After the compose has finished it will get synced to /mnt/koji/atomic/26/ which corresponds
to the public URL https://kojipkgs.fedoraproject.org/atomic/26/.
During the compose, once the ostree commit has been created, the installer ISO and the cloud images can get created. The definition for the installer ISO and the cloud images are all within the same Pungi config.
The LEGOs
There are a few building blocks we need to talk about before we really jump into how this whole thing is put together. First, we have to talk about how the OSTree gets created and then the artifacts that are derived from it.
The RPM OSTree
To create the OSTree Pungi ends up calling out to a Koji runroot task. This runroot task essentially runs a command and harvests the results. The command that eventually gets run is an rpm-ostree command that looks something like this:
rpm-ostree compose tree --repo=/mnt/koji/compose/atomic/26/ \
--write-commitid-to=/mnt/koji/compose/branched/Fedora-26-20170705.n.0/logs/x86_64/Atomic/ostree-2/commitid.log \
/mnt/koji/compose/branched/Fedora-26-20170705.n.0/work/ostree-2/config_repo/fedora-atomic-docker-host.json
From the ostree part of the Pungi config you can see where some of the arguments for this command came from:
ostree = [
("^Atomic$", {
"x86_64": {
"treefile": "fedora-atomic-docker-host.json",
"config_url": "https://pagure.io/fedora-atomic.git",
"config_branch": "f26",
"source_repo_from": "Everything",
"ostree_repo": "/mnt/koji/compose/atomic/26/",
'failable': ['*'],
}
}),
We store our inputs inputs to rpm-ostree compose tree in
the fedora-atomic git repo. The
fedora-26.repo
file defines the dnf repositories and the
fedora-atomic-host.json
file defines which of those repositories to use, which rpms to pull
from them, and a few other things.
The Installer ISO
For the installer ISO Pungi again calls out to a Koji runroot task. The command that eventually gets run is a lorax command that looks something like this:
lorax --product=Fedora --version=26 --release=20170705.n.0 \
--source=http://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170705.n.0/compose/Everything/x86_64/os \
--variant=Atomic --nomacboot --volid=Fedora-Atomic-ostree-x86_64-26 \
--installpkgs=fedora-productimg-atomic \
--add-template=/mnt/koji/compose/branched/Fedora-26-20170705.n.0/work/x86_64/Atomic/lorax_templates/ostree-based-installer/lorax-configure-repo.tmpl \
--add-template=/mnt/koji/compose/branched/Fedora-26-20170705.n.0/work/x86_64/Atomic/lorax_templates/ostree-based-installer/lorax-embed-repo.tmpl \
--add-template-var=ostree_install_repo=https://kojipkgs.fedoraproject.org/compose/atomic/26/ \
--add-template-var=ostree_update_repo=https://kojipkgs.fedoraproject.org/atomic/26/ \
--add-template-var=ostree_osname=fedora-atomic \
--add-template-var=ostree_install_ref=fedora/26/x86_64/atomic-host \
--add-template-var=ostree_update_ref=fedora/26/x86_64/atomic-host \
--logfile=/mnt/koji/compose/branched/Fedora-26-20170705.n.0/logs/x86_64/Atomic/ostree_installer-1/lorax.log \
--rootfs-size=3 /mnt/koji/compose/branched/Fedora-26-20170705.n.0/work/x86_64/Atomic/ostree_installer
From the installer ISO part of the Pungi config you can see some configuration that was set and eventually translated into the lorax command above.
ostree_installer = [
("^Atomic$", {
"x86_64": {
"source_repo_from": "Everything",
"release": None,
"rootfs_size": "3",
"installpkgs": ["fedora-productimg-atomic"],
"add_template": ["ostree-based-installer/lorax-configure-repo.tmpl",
"ostree-based-installer/lorax-embed-repo.tmpl"],
"add_template_var": [
"ostree_install_repo=https://kojipkgs.fedoraproject.org/compose/atomic/26/",
"ostree_update_repo=https://kojipkgs.fedoraproject.org/atomic/26/",
"ostree_osname=fedora-atomic",
"ostree_install_ref=fedora/26/x86_64/atomic-host",
"ostree_update_ref=fedora/26/x86_64/atomic-host",
],
'template_repo': 'https://pagure.io/fedora-lorax-templates.git',
'template_branch': 'f26',
'failable': ['*'],
}
}),
There are a few lorax templates we are passing in as well as some variables to those templates. The templates are stored in the fedora-lorax-templates git repo.
The Cloud Images
For the cloud images Koji has higher level support for building them than it does for the installer ISO or for creating the ostree. It doesn’t need to run things in a runroot, which is generic, but rather it can create an ImageBuild task to create an image.
This is still all defined in the Pungi config. The image-build sections
for Atomic Host are here
in the Pungi config. One of these sections looks like:
'image-build': {
'format': [('qcow2','qcow2'), ('raw-xz','raw.xz')],
'name': 'Fedora-Atomic',
'kickstart': 'fedora-atomic.ks',
'distro': 'Fedora-22',
'disk_size': 6,
'arches': ['x86_64'],
'install_tree_from': 'Cloud',
'subvariant': 'Atomic',
'failable': ['*'],
}
You’ll notice that we say what kickstart file to use, but we don’t define where to pull the kickstart file from. The git repo for the kickstarts is defined along with a few other varialbes earlier in the file.
After Fedora Major Release: Bodhi + Pungi + Release
Once Fedora is officially released for a particular version there is a release day yum/dnf repository that is made from the rpms that were stable on the day of release. This repository is frozen and will not change.
There are two more repositories that become significant now. These are
the updates and the updates-testing repositories. The updates
repo contains packages that have passed testing and are available to Fedora users
whenever they run dnf update. The updates-testing repo is for
packages that have been built and have been submitted as an update for
people to test to make sure nothing is broken before graduating to updates.
The updates-testing repo is not enabled by default. A user would
have to willingly enable it for the purpose of doing testing.
All of that is to say that there are other repos that exist after release day that are for updated rpms.
Bodhi
Within Fedora there is a tool known as Bodhi that is responsible for tracking what state particular packages are in and moving them between the updates and updates-testing repos. As part of this, Bodhi is currently responsible for creating the repositories and also the OSTree commits from the new content that it just created a repo for. Bodhi was the most logical candidate for this at the time it was implemented because we wanted to create a new commit as soon as we could grab new content (right after the repositories are created).
After creating the OSTree content it gets synced to fedora/26/x86_64/updates/atomic-host
ref within the OSTree repository at https://kojipkgs.fedoraproject.org/atomic/26/.
This is a global location/ref that users could pull from for existing
installed systems.
Pungi
When Bodhi runs and creates a new updates repo it also creates a
new OSTree. When we do an official two week release we release other
artifacts (ISOs, Qcows, etc) too. These get built once a day and uses the
fedora-atomic.conf
Pungi config (which gets called from the
twoweek-nightly.sh
script). Basically the config creates repos from the f26-atomic Koji tag
which inherits from the f26 tag (release day tag). It creates some
repos from rpms in the Cloud variant and then builds installer
images and qcows based on those rpms. The installer images and the
qcows actually pull the OSTree from the commit that was generated
by the last Bodhi run, though. So the yum/dnf repository created
during the Pungi run only affects the installer, not the installed
OSTree in the installed systems.
Performing A Two Week Atomic Host Release
So with the OSTree created by Bodhi and the other artifacts created by
Pungi we can now test do an official two week Atomic Host release.
This is currently run by a member of releng and the process is
documented by this
Standard Operating Procedure guide. It eventually tells the operator
to run this releng script
to do the release. The script does several things but most noteworthy
is updating the fedora/26/x86_64/atomic-host ref within the OSTree
repo, syncing out the ISOs/qcows to the mirrors, and updating the
website.
Conclusion
Our processes differ slightly for before and after Fedora release. The entire pipeline involves Koji, Bodhi, and Pungi, and many tools underneath the covers. There is a lot of working being done to try to improve these processes including calling Pungi from Bodhi in the short term and possibly building Atomic Host from a Module in the longer term. Please reach out to us with any questions about this whole process in #atomic on freenode.