How Do We Create OSTree Repos and Artifacts in Fedora

a more permanent version of this content lives here

Introduction

NOTE: For background on OSTree check out the docs.

When you want to create a new OSTree using rpm-ostree you usually define a few yum repos, and then a json file that says what rpms you want to be composed in the tree. You then run an rpm-ostree compose tree command to create the commit in the ostree repo. Once the ostree commit has been created you can then create installer images (ISOs) and cloud/VM images (qcow, etc) from that ostree.

How does Fedora do this? It’s a bit complicated, but I’ll try to cover the bases.

First I’ll give you a little high level view of the way things are built before major release, all using Pungi. I’ll then try to dig into The LEGOs that are used to build the artifacts we care about. Finally, I’ll try to explain how things are different after a major release is out and how we end up getting updated rpms and eventually two week releases out the door.

Before Fedora Major Release: Building Everything Using Pungi

Prior to any major release of Fedora the release engineering team builds the whole world every night. This is done using a tool called Pungi. A script is called to kick off the nightly run via a cron job. The Pungi config that is used is the fedora.conf file from the pungi-fedora git repo.

For Fedora 26 you can see where the ostree for Atomic Host is defined in the Pungi config, here, in the ostree section. During a compose you can see that the ostree will get placed into "ostree_repo": "/mnt/koji/compose/atomic/26/",. After the compose has finished it will get synced to /mnt/koji/atomic/26/ which corresponds to the public URL https://kojipkgs.fedoraproject.org/atomic/26/.

During the compose, once the ostree commit has been created, the installer ISO and the cloud images can get created. The definition for the installer ISO and the cloud images are all within the same Pungi config.

The LEGOs

There are a few building blocks we need to talk about before we really jump into how this whole thing is put together. First, we have to talk about how the OSTree gets created and then the artifacts that are derived from it.

The RPM OSTree

To create the OSTree Pungi ends up calling out to a Koji runroot task. This runroot task essentially runs a command and harvests the results. The command that eventually gets run is an rpm-ostree command that looks something like this:

rpm-ostree compose tree --repo=/mnt/koji/compose/atomic/26/ \
--write-commitid-to=/mnt/koji/compose/branched/Fedora-26-20170705.n.0/logs/x86_64/Atomic/ostree-2/commitid.log \
/mnt/koji/compose/branched/Fedora-26-20170705.n.0/work/ostree-2/config_repo/fedora-atomic-docker-host.json

From the ostree part of the Pungi config you can see where some of the arguments for this command came from:

ostree = [
    ("^Atomic$", {
        "x86_64": {
            "treefile": "fedora-atomic-docker-host.json",
            "config_url": "https://pagure.io/fedora-atomic.git",
            "config_branch": "f26",
            "source_repo_from": "Everything",
            "ostree_repo": "/mnt/koji/compose/atomic/26/",
            'failable': ['*'],
        }
    }),

We store our inputs inputs to rpm-ostree compose tree in the fedora-atomic git repo. The fedora-26.repo file defines the dnf repositories and the fedora-atomic-host.json file defines which of those repositories to use, which rpms to pull from them, and a few other things.

The Installer ISO

For the installer ISO Pungi again calls out to a Koji runroot task. The command that eventually gets run is a lorax command that looks something like this:

lorax --product=Fedora --version=26 --release=20170705.n.0 \
--source=http://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170705.n.0/compose/Everything/x86_64/os \
--variant=Atomic --nomacboot --volid=Fedora-Atomic-ostree-x86_64-26 \
--installpkgs=fedora-productimg-atomic \
--add-template=/mnt/koji/compose/branched/Fedora-26-20170705.n.0/work/x86_64/Atomic/lorax_templates/ostree-based-installer/lorax-configure-repo.tmpl \
--add-template=/mnt/koji/compose/branched/Fedora-26-20170705.n.0/work/x86_64/Atomic/lorax_templates/ostree-based-installer/lorax-embed-repo.tmpl \
--add-template-var=ostree_install_repo=https://kojipkgs.fedoraproject.org/compose/atomic/26/ \
--add-template-var=ostree_update_repo=https://kojipkgs.fedoraproject.org/atomic/26/ \
--add-template-var=ostree_osname=fedora-atomic \
--add-template-var=ostree_install_ref=fedora/26/x86_64/atomic-host \
--add-template-var=ostree_update_ref=fedora/26/x86_64/atomic-host \
--logfile=/mnt/koji/compose/branched/Fedora-26-20170705.n.0/logs/x86_64/Atomic/ostree_installer-1/lorax.log \
--rootfs-size=3 /mnt/koji/compose/branched/Fedora-26-20170705.n.0/work/x86_64/Atomic/ostree_installer

From the installer ISO part of the Pungi config you can see some configuration that was set and eventually translated into the lorax command above.

ostree_installer = [
    ("^Atomic$", {
        "x86_64": {
            "source_repo_from": "Everything",
            "release": None,
            "rootfs_size": "3",
            "installpkgs": ["fedora-productimg-atomic"],
            "add_template": ["ostree-based-installer/lorax-configure-repo.tmpl",
                             "ostree-based-installer/lorax-embed-repo.tmpl"],
            "add_template_var": [
                "ostree_install_repo=https://kojipkgs.fedoraproject.org/compose/atomic/26/",
                "ostree_update_repo=https://kojipkgs.fedoraproject.org/atomic/26/",
                "ostree_osname=fedora-atomic",
                "ostree_install_ref=fedora/26/x86_64/atomic-host",
                "ostree_update_ref=fedora/26/x86_64/atomic-host",
            ],
            'template_repo': 'https://pagure.io/fedora-lorax-templates.git',
            'template_branch': 'f26',
            'failable': ['*'],
        }
    }),

There are a few lorax templates we are passing in as well as some variables to those templates. The templates are stored in the fedora-lorax-templates git repo.

The Cloud Images

For the cloud images Koji has higher level support for building them than it does for the installer ISO or for creating the ostree. It doesn’t need to run things in a runroot, which is generic, but rather it can create an ImageBuild task to create an image.

This is still all defined in the Pungi config. The image-build sections for Atomic Host are here in the Pungi config. One of these sections looks like:

	'image-build': {
			'format': [('qcow2','qcow2'), ('raw-xz','raw.xz')],
			'name': 'Fedora-Atomic',
			'kickstart': 'fedora-atomic.ks',
			'distro': 'Fedora-22',
			'disk_size': 6,
			'arches': ['x86_64'],
			'install_tree_from': 'Cloud',
			'subvariant': 'Atomic',
			'failable': ['*'],
	}

You’ll notice that we say what kickstart file to use, but we don’t define where to pull the kickstart file from. The git repo for the kickstarts is defined along with a few other varialbes earlier in the file.

After Fedora Major Release: Bodhi + Pungi + Release

Once Fedora is officially released for a particular version there is a release day yum/dnf repository that is made from the rpms that were stable on the day of release. This repository is frozen and will not change.

There are two more repositories that become significant now. These are the updates and the updates-testing repositories. The updates repo contains packages that have passed testing and are available to Fedora users whenever they run dnf update. The updates-testing repo is for packages that have been built and have been submitted as an update for people to test to make sure nothing is broken before graduating to updates. The updates-testing repo is not enabled by default. A user would have to willingly enable it for the purpose of doing testing.

All of that is to say that there are other repos that exist after release day that are for updated rpms.

Bodhi

Within Fedora there is a tool known as Bodhi that is responsible for tracking what state particular packages are in and moving them between the updates and updates-testing repos. As part of this, Bodhi is currently responsible for creating the repositories and also the OSTree commits from the new content that it just created a repo for. Bodhi was the most logical candidate for this at the time it was implemented because we wanted to create a new commit as soon as we could grab new content (right after the repositories are created).

After creating the OSTree content it gets synced to fedora/26/x86_64/updates/atomic-host ref within the OSTree repository at https://kojipkgs.fedoraproject.org/atomic/26/. This is a global location/ref that users could pull from for existing installed systems.

Pungi

When Bodhi runs and creates a new updates repo it also creates a new OSTree. When we do an official two week release we release other artifacts (ISOs, Qcows, etc) too. These get built once a day and uses the fedora-atomic.conf Pungi config (which gets called from the twoweek-nightly.sh script). Basically the config creates repos from the f26-atomic Koji tag which inherits from the f26 tag (release day tag). It creates some repos from rpms in the Cloud variant and then builds installer images and qcows based on those rpms. The installer images and the qcows actually pull the OSTree from the commit that was generated by the last Bodhi run, though. So the yum/dnf repository created during the Pungi run only affects the installer, not the installed OSTree in the installed systems.

Performing A Two Week Atomic Host Release

So with the OSTree created by Bodhi and the other artifacts created by Pungi we can now test do an official two week Atomic Host release. This is currently run by a member of releng and the process is documented by this Standard Operating Procedure guide. It eventually tells the operator to run this releng script to do the release. The script does several things but most noteworthy is updating the fedora/26/x86_64/atomic-host ref within the OSTree repo, syncing out the ISOs/qcows to the mirrors, and updating the website.

Conclusion

Our processes differ slightly for before and after Fedora release. The entire pipeline involves Koji, Bodhi, and Pungi, and many tools underneath the covers. There is a lot of working being done to try to improve these processes including calling Pungi from Bodhi in the short term and possibly building Atomic Host from a Module in the longer term. Please reach out to us with any questions about this whole process in #atomic on freenode.