In Part 1 of this series we learned a little about the technology behind Atomic Host and how to interact with a deployed system. In this section we will cover the Configuring Storage for Containers topic from the outline in Part 0.
History of Container Storage
One of the early goals of Atomic Host was to be a good platform for
running containerized workloads. This is still a fundamental goal of
Atomic Host and certainly includes making sure that the container
runtime (currently the
docker daemon) has proper storage
configuration such that it can get a balance of good performance and
For a long while the best option for Red Hat based operating systems
was to use the
devicemapper storage backend for docker. The benefits
were that the devicemapper backend was stable, decently performant,
and had SELinux support. More recently, the
overlay2 backend has
emerged as a legitimate option because SELinux support has landed
and there are significant performance improvements over devicemapper.
However, most of the storage needs and original designs were centered around devicemapper and the fact that the backend requires there to be a separate dedicated block device that can be donated to the dm thin-pool. This could be an actual physical block device, or a LVM logical volume from an existing volume group.
The container-storage-setup Tool
The need for a device to be dedicated to container storage prompted
the need for a tool, called container-storage-setup
(previously known as
docker-storage-setup) to make this easy for spinning up
instances of a container host. Atomic Host has conveniently shipped
with extra space available on pre-built disk images so that
container-storage-setup could use it when configuring storage for
container-storage-setup gets executed automatically by
systemd when the
docker.service starts, leveraging the tool is usually
as simple as populating the
early in boot (i.e. cloud-init or some other mechanism)
with some config values that tell the utility what to do.
In order to investigate some of the options we have available take a look at the existing contents of that file on the Atomic Host:
[root@localhost ~]# cat /etc/sysconfig/docker-storage-setup # Edit this file to override any configuration options specified in # /usr/share/container-storage-setup/container-storage-setup. # # For more details refer to "man container-storage-setup" STORAGE_DRIVER=overlay2 CONTAINER_ROOT_LV_NAME=docker-root-lv CONTAINER_ROOT_LV_MOUNT_PATH=/var/lib/docker GROWPART=true
As you can see from the contents of the file, in Fedora 26 we
have already moved on to setting overlay2 as the default storage
driver for the container runtime. In this case we are also defauting
to creating a new LV (
docker-root-lv) and a filesystem (XFS) on top of
that LV to mount on
/var/lib/docker. The overlay2 backend will then
run on top of that filesystem.
NOTE: There are a lot of options that can be provided to the
container-storage-setup tool. You can investigate them by checking
Inspecting The Booted System
We just saw the contents of the
file. Let’s check out the output from the
to see what happened when the system came up and the docker daemon was
[root@localhost ~]# systemctl status docker-storage-setup.service -o cat ● docker-storage-setup.service - Docker Storage Setup Loaded: loaded (/usr/lib/systemd/system/docker-storage-setup.service; enabled; vendor preset: disabled) Active: inactive (dead) since Mon 2017-08-28 00:15:50 UTC; 27min ago Main PID: 720 (code=exited, status=0/SUCCESS) Starting Docker Storage Setup... CHANGED: partition=2 start=616448 old: size=83269632 end=83886080 new: size=85366784,end=85983232 Physical volume "/dev/vda2" changed 1 physical volume(s) resized / 0 physical volume(s) not resized Logical volume "docker-root-lv" created. Started Docker Storage Setup.
Looks like the root partition was extended and the
was created. Although we don’t explicitly see the output here it is
also true that an XFS filesystem was created on top of that and
storage options that specify
overlay2 were added to the
/etc/sysconfig/docker-storage configuration file.
We can see the LV is mounted on
[root@localhost ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 252:0 0 41G 0 disk ├─vda1 252:1 0 300M 0 part /boot └─vda2 252:2 0 40.7G 0 part ├─atomicos-root 253:0 0 13G 0 lvm /sysroot └─atomicos-docker--root--lv 253:1 0 15.1G 0 lvm /sysroot/ostree/deploy/fedora-atomic/var/lib/docker
Also, if we query
docker we can see that it is using the
[root@localhost ~]# docker info 2>/dev/null | grep 'Storage Driver' Storage Driver: overlay2
Changing Storage Configuration
NOTE: If you are going to change container storage on a system
please be mindful that you will lose the container
images/data that currently exist on your system. The Atomic
atomic storage import/export to attempt to save
and restore containers for this type of scenario.
We can also use the
container-storage-setup utility to change the
storage configuration on a system. Again, it is suggested that you
look at the
for an explanation of options that you can put in the
/etc/sysconfig/docker-storage-setup configuration file.
For example, if you want to switch from
you would need to change it so that
As an exercise, let’s switch this system to the devicemapper backend.
We’ll first stop docker, unmount the filesystem which was backing the overlay2 driver, and also remove the logical volume that filesystem was build on top of.
[root@localhost ~]# systemctl stop docker [root@localhost ~]# umount /var/lib/docker [root@localhost ~]# lvremove /dev/atomicos/docker-root-lv Do you really want to remove active logical volume atomicos/docker-root-lv? [y/n]: y Logical volume "docker-root-lv" successfully removed
Since we just unmounted and removed the LV behind
we need to remove the systemd mount unit that was configured
/var/lib/docker. Removing this file is like removing an
entry from the
[root@localhost ~]# rm /etc/systemd/system/docker-storage-setup.service.wants/var-lib-docker.mount [root@localhost ~]# systemctl daemon-reload
Next we’ll remove the
/etc/sysconfig/docker-storage file, which
has options to give to the docker daemon to tell it what storage
driver to use, and also we’ll overwrite all settings in the
/etc/sysconfig/docker-storage-setup so that
will know we want to use the
devicemapper backend. We’ll then run
container-storage-setup to create the devicemapper thin pool
and populate a new
/etc/sysconfig/docker-storage with options for
the docker daemon.
NOTE: Some or all of these operations may be able to be done by
atomic storage modify command. Feel free to
investigate that option.
[root@localhost ~]# rm /etc/sysconfig/docker-storage [root@localhost ~]# echo 'STORAGE_DRIVER=devicemapper' > /etc/sysconfig/docker-storage-setup [root@localhost ~]# /usr/bin/container-storage-setup Using default stripesize 64.00 KiB. Rounding up size to full physical extent 44.00 MiB Logical volume "docker-pool" created. Logical volume atomicos/docker-pool changed.
Now that this is done you can see thin LV that was set up in the
[root@localhost ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 252:0 0 41G 0 disk ├─vda1 252:1 0 300M 0 part /boot └─vda2 252:2 0 40.7G 0 part ├─atomicos-root 253:0 0 13G 0 lvm /sysroot ├─atomicos-docker--pool_tmeta 253:1 0 44M 0 lvm │ └─atomicos-docker--pool 253:3 0 11G 0 lvm └─atomicos-docker--pool_tdata 253:2 0 11G 0 lvm └─atomicos-docker--pool 253:3 0 11G 0 lvm [root@localhost ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert docker-pool atomicos twi-a-t--- 11.02g 0.00 0.09 root atomicos -wi-ao---- 12.93g
Now we can start docker and verify that the devicemapper backend is being used:
[root@localhost ~]# systemctl start docker [root@localhost ~]# docker info 2>/dev/null | grep 'Storage Driver' Storage Driver: devicemapper
To cap things off let’s import a container image from our lab files and run a container to see that new devicemapper objects are getting created for this storage:
[root@localhost ~]# atomic storage import --dir /srv/localweb/containers/ Importing image: a16c8800bb14 7d4769f4070d: Loading layer [==================================================>] 242.5 MB/242.5 MB 25ca5f0393cd: Loading layer [==================================================>] 359 MB/359 MB 960139190bea: Loading layer [==================================================>] 71.77 MB/71.77 MB Loaded image: registry.fedoraproject.org/f25/httpd:latest Importing volumes atomic import completed successfully Would you like to cleanup (rm -rf /srv/localweb/containers/) the temporary directory [y/N]N Please restart docker daemon for the changes to take effect [root@localhost ~]# [root@localhost ~]# docker run -d a16c8800bb14 sleep 600 b0509ea6c6b6811915f3f2d15a9fd344836f628a638d797683e7b27c4be84205 [root@localhost ~]# [root@localhost ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 252:0 0 41G 0 disk ├─vda1 252:1 0 300M 0 part /boot └─vda2 252:2 0 40.7G 0 part ├─atomicos-root 253:0 0 13G 0 lvm /sysroot ├─atomicos-docker--pool_tmeta 253:1 0 44M 0 lvm │ └─atomicos-docker--pool 253:3 0 11G 0 lvm │ └─docker-253:0-14918141-d748db4190b91666509effaabaa06c24d6e5ed836e5421ce914250e7eb1ac5a0 │ 253:4 0 10G 0 dm /var/lib/docker/devicemapper/mnt/d748db4190b916 └─atomicos-docker--pool_tdata 253:2 0 11G 0 lvm └─atomicos-docker--pool 253:3 0 11G 0 lvm └─docker-253:0-14918141-d748db4190b91666509effaabaa06c24d6e5ed836e5421ce914250e7eb1ac5a0 253:4 0 10G 0 dm /var/lib/docker/devicemapper/mnt/d748db4190b916
As the container runs (
sleep 600) we can see the new devicemapper
mounts that were created for the container in the
Part 2 Wrap Up
Part 2 of this lab has introduced us to Container Storage and the
necessity for the
container-storage-setup tool. We also inspected
how the storage gets configured on an Atomic Host and switched the
storage from one storage backend to another. In the
we’ll cover rebase, upgrade and rollback for Atomic Host as well as some
basic methods for viewing OSTree commit history and inspecting changes
to a running system.