Introduction
In Fedora we are moving to a unified OSTree repo structure where there is a single OSTree repository that is the remote for all branches of Fedora (rawhide, branched, stable, etc). As part of this we want to be able to define different retention policies for different branches within the repository. For rawhide we’ll retain a few weeks worth of commits, but for stable we don’t want to prune any of the commits.
Allowing for this flexibility required an RFE
against OSTree upstream and it was implemented
by adding the --only-branch
and --retain-branch-depth
options
to ostree prune
. Let’s give them a test drive.
Setting up a test repo
We’ll create a new test repo and then pull OSTree content for the
last 11 commits (--depth=10
) from a few refs.
[dustymabe@host ~]$ mkdir repo
[dustymabe@host tmp]$ cd repo
[dustymabe@host repo]$ ostree init --repo=./
[dustymabe@host repo]$ ostree remote add f27 https://kojipkgs.fedoraproject.org/atomic/27/ --no-gpg-verify
[dustymabe@host repo]$
[dustymabe@host repo]$ sudo ostree pull f27:fedora/27/x86_64/atomic-host --depth=10
5688 metadata, 34013 content objects fetched; 1338909 KiB transferred in 71 seconds
[dustymabe@host repo]$
[dustymabe@host repo]$ sudo ostree pull f27:fedora/27/aarch64/atomic-host --depth=10
3189 metadata, 14266 content objects fetched; 788984 KiB transferred in 43 seconds
[dustymabe@host repo]$
[dustymabe@host repo]$ sudo ostree pull f27:fedora/27/ppc64le/atomic-host --depth=10
3012 metadata, 12942 content objects fetched; 842778 KiB transferred in 51 seconds
[dustymabe@host repo]$
Now we have 3 refs in the repo, one for x86_64
, aarch64
, and ppc64le
.
[dustymabe@host repo]$ ostree refs
f27:fedora/27/ppc64le/atomic-host
f27:fedora/27/x86_64/atomic-host
f27:fedora/27/aarch64/atomic-host
Each of these refs have 11 commits in the history (10 + the commit the ref is currently pointing to). We can show this by using a short script.
[dustymabe@host repo]$ cat count.sh
#!/bin/bash
for ref in $(ostree refs); do
echo -en "$ref\t"
ostree log "$ref" --raw | grep '^commit' | wc -l
done
[dustymabe@host repo]$ bash count.sh
f27:fedora/27/ppc64le/atomic-host 11
f27:fedora/27/x86_64/atomic-host 11
f27:fedora/27/aarch64/atomic-host 11
Pruning content from a single ref
We’ll start by pruning 2 commits from the ppc64le
branch within the repo.
Before we actually prune let’s do a dry run using --no-prune
to get an idea of
what would happen:
[dustymabe@host repo]$ sudo ostree prune --only-branch f27:fedora/27/ppc64le/atomic-host --depth=8 --no-prune
Total objects: 73077
Would delete: 3415 objects, freeing 0 bytes
[dustymabe@host repo]$
[dustymabe@host repo]$ bash count.sh
f27:fedora/27/ppc64le/atomic-host 11
f27:fedora/27/x86_64/atomic-host 11
f27:fedora/27/aarch64/atomic-host 11
So we Would delete: 3415 objects
from the repo if we ran this prune operation.
You can see from the count script that we didn’t actually prune anything and
all refs still have 11 commits in the repo. Let’s actually prune now:
[dustymabe@host repo]$ sudo ostree prune --only-branch f27:fedora/27/ppc64le/atomic-host --depth=8
Total objects: 73077
Deleted 3415 objects, 242.0 MB freed
[dustymabe@host repo]$
[dustymabe@host repo]$ bash count.sh
f27:fedora/27/ppc64le/atomic-host 9
f27:fedora/27/x86_64/atomic-host 11
f27:fedora/27/aarch64/atomic-host 11
In this case we pruned those 3415 objects and freed up 242 MB in the process.
We can see from the count script that we are down to 9 commits in the repo
for the ppc64le
ref.
Pruning different branches at different rates
In this case we want to prune different branches at differing rates.
We can either run two separate commands for this or we can use
the --retain-branch-depth
option.
[dustymabe@host repo]$ sudo ostree prune --only-branch f27:fedora/27/x86_64/atomic-host --only-branch f27:fedora/27/aarch64/atomic-host --depth=6 --retain-branch-depth f27:fedora/27/x86_64/atomic-host=7
Total objects: 69662
Deleted 8100 objects, 604.5 MB freed
[dustymabe@host repo]$ bash count.sh
f27:fedora/27/ppc64le/atomic-host 9
f27:fedora/27/x86_64/atomic-host 8
f27:fedora/27/aarch64/atomic-host 7
In the above example we pruned the aarch64
ref down to 7 commits, but we
only wanted to prune the x86_64
ref to 8 commits. To do this we defined
a specific policy for the x86_64
branch using
--retain-branch-depth f27:fedora/27/x86_64/atomic-host=7
. We can see from
the count.sh
script that the counts are at 9, 8, and 7 for our various
refs.
Pruning based on time
We don’t only have to prune based on depth. We can also use the
--keep-younger-than
option to prune specific branches:
[dustymabe@host repo]$ sudo ostree prune --keep-younger-than="10 days ago" --only-branch f27:fedora/27/ppc64le/atomic-host
Total objects: 61562
Deleted 5130 objects, 525.6 MB freed
[dustymabe@host repo]$
[dustymabe@host repo]$ bash count.sh
f27:fedora/27/ppc64le/atomic-host 3
f27:fedora/27/x86_64/atomic-host 8
f27:fedora/27/aarch64/atomic-host 7
In this case we kept all commits that were younger than 10 days ago
.
This brought us down to 3 commits for the ppc64le
branch.
Fin
Now we know how to define/execute different policies for different branches in an OSTree repo. This will help us in Fedora manage the size of our repositories.
NOTE: In the future we may be delivering OSTree content via a special RPM using RPM-OSTree rojig instead of hosting an OSTree repository. Stay tuned for updates on that.