Archive for the 'Anaconda' Category

Fedora BTRFS+Snapper PART 1: System Preparation

The Problem

For some time now I have wanted a linux desktop setup where I could run updates automatically and not worry about losing productivity if my system gets hosed from the update. My desired setup to achieve this has been a combination of snapper and BTRFS, but unfortunately the support on Fedora for full rollback isn't quite there.

In Fedora 22 the support for rollback was added but there is one final piece of the puzzle that is missing that I need in order to have a fully working setup: I needed GRUB to respect the default subvolume that is set on the BTRFS filesystem. In the past GRUB did use the default subvolume but this behavior was removed in 82591fa (link).

With GRUB respecting the default subvolume I can include /boot/ just as a directory on my system (not as a separate subvolume) and it will be included in all of the snapshots that are created by snapper of the root filesystem.

In order to get this functionality I grabbed some of the patches from the SUSE guys and applied them to the Fedora GRUB rpm. All of the work and the resulting rpms can be found here.

System Preparation

So now I had a GRUB rpm that would work for me. The first step is to get my system up and running in a setup that I could then use snapper on top of. I mentioned before that I wanted to put /boot/ just as a directory on the BTRFS filesystem. I also wanted it to be encrypted as I have done in the past.

This means I have yet another setup that is funky and I'll need to basically install it from scratch using Anaconda and a chroot environment.

After getting up and running in anaconda I then switched to a different virtual terminal and formatted my hard disk, set up an encrypted LUKS device, created a VG and two LVs, and finally a BTRFS filesystem:

[anaconda root@localhost ~]# fdisk /dev/sda <<EOF
o
n
p
1
2048

w
EOF
[anaconda root@localhost ~]# lsblk /dev/sda
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 465.8G  0 disk
`-sda1   8:1    0 465.8G  0 part
[anaconda root@localhost ~]# cryptsetup luksFormat /dev/sda1
[anaconda root@localhost ~]# cryptsetup luksOpen /dev/sda1 cryptodisk
[anaconda root@localhost ~]# vgcreate vgroot /dev/mapper/cryptodisk
[anaconda root@localhost ~]# lvcreate --size=4G --name lvswap vgroot
[anaconda root@localhost ~]# mkswap /dev/vgroot/lvswap
[anaconda root@localhost ~]# lvcreate -l 100%FREE --name lvroot vgroot
[anaconda root@localhost ~]# mkfs.btrfs /dev/vgroot/lvroot

NOTE: Most of the commands run above have truncated output for brevity.

The next step was to mount the filesystem and install software into the filesystem in a chrooted environment. Since the dnf binary isn't actually installed in the anaconda environment by default we first need to install it:

[anaconda root@localhost ~]# rpm -ivh --nodeps /run/install/repo/Packages/d/dnf-1.0.0-1.fc22.noarch.rpm
warning: /run/install/repo/Packages/d/dnf-1.0.0-1.fc22.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 8e1431d5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:dnf-1.0.0-1.fc22                 ################################# [100%]

Now we can "create" a repo file from the repo that is on the media and install the bare minimum (the filesystem rpm):

[anaconda root@localhost ~]# mount /dev/vgroot/lvroot /mnt/sysimage/
[anaconda root@localhost ~]# mkdir /etc/yum.repos.d
[anaconda root@localhost ~]# cat <<EOF > /etc/yum.repos.d/dvd.repo
[dvd]
name=dvd
baseurl=file:///run/install/repo
enabled=1
gpgcheck=0
EOF
[anaconda root@localhost ~]# dnf install -y --releasever=22 --installroot=/mnt/sysimage filesystem
...
Complete!

The reason we only installed the filesystem rpm is because a lot of the other rpms we are going to install will fail if some of the "special" directories aren't mounted. We'll go ahead and mount them now:

[anaconda root@localhost ~]# mount -v -o bind /dev /mnt/sysimage/dev/
mount: /dev bound on /mnt/sysimage/dev.
[anaconda root@localhost ~]# mount -v -o bind /run /mnt/sysimage/run/
mount: /run bound on /mnt/sysimage/run.
[anaconda root@localhost ~]# mount -v -t proc proc /mnt/sysimage/proc/
mount: proc mounted on /mnt/sysimage/proc.
[anaconda root@localhost ~]# mount -v -t sysfs sys /mnt/sysimage/sys/
mount: sys mounted on /mnt/sysimage/sys.

Now we can install the rest of the software into the chroot environment:

[anaconda root@localhost ~]# cp /etc/yum.repos.d/dvd.repo /mnt/sysimage/etc/yum.repos.d/
[anaconda root@localhost ~]# dnf install -y --installroot=/mnt/sysimage --disablerepo=* --enablerepo=dvd @core @standard kernel btrfs-progs lvm2
...
Complete!

We can also install the "special" GRUB packages that I created and then get rid of the repo file because we won't need it any longer:

[anaconda root@localhost ~]# dnf install -y --installroot=/mnt/sysimage --disablerepo=* --enablerepo=dvd 
https://github.com/dustymabe/fedora-grub-boot-btrfs-default-subvolume/raw/master/rpmbuild/RPMS/x86_64/grub2-2.02-0.16.fc22.dusty.x86_64.rpm 
https://github.com/dustymabe/fedora-grub-boot-btrfs-default-subvolume/raw/master/rpmbuild/RPMS/x86_64/grub2-tools-2.02-0.16.fc22.dusty.x86_64.rpm
...
Complete!
[anaconda root@localhost ~]# rm /mnt/sysimage/etc/yum.repos.d/dvd.repo

Now we can do some minimal system configuration by chrooting into the system and setting up crypttab, setting up fstab, setting the root password and setting up the system to a relabel on boot:

[anaconda root@localhost ~]# chroot /mnt/sysimage
[anaconda root@localhost /]# ls -l /dev/disk/by-uuid/f0d889d8-5225-4d9d-9a89-edd387e65ab7
lrwxrwxrwx. 1 root root 10 Jul 14 02:24 /dev/disk/by-uuid/f0d889d8-5225-4d9d-9a89-edd387e65ab7 -> ../../sda1
[anaconda root@localhost /]# cat <<EOF > /etc/crypttab
cryptodisk /dev/disk/by-uuid/f0d889d8-5225-4d9d-9a89-edd387e65ab7 -
EOF
[anaconda root@localhost /]# cat <<EOF > /etc/fstab
/dev/vgroot/lvroot / btrfs defaults 1 1
/dev/vgroot/lvswap swap swap defaults 0 0
EOF
[anaconda root@localhost /]# passwd --stdin root <<< "password"
Changing password for user root.
passwd: all authentication tokens updated successfully.
[anaconda root@localhost /]# touch /.autorelabel

Finally configure and install GRUB on sda and generate a ramdisk that has all the required modules using dracut:

[anaconda root@localhost /]# echo GRUB_ENABLE_CRYPTODISK=y >> /etc/default/grub
[anaconda root@localhost /]# echo SUSE_BTRFS_SNAPSHOT_BOOTING=true >> /etc/default/grub
[anaconda root@localhost /]# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
File descriptor 4 (/) leaked on vgs invocation. Parent PID 29465: /usr/sbin/grub2-probe
File descriptor 4 (/) leaked on vgs invocation. Parent PID 29465: /usr/sbin/grub2-probe
Found linux image: /boot/vmlinuz-4.0.4-301.fc22.x86_64
Found initrd image: /boot/initramfs-4.0.4-301.fc22.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-225efda374c043e3886d349ef724c79e
Found initrd image: /boot/initramfs-0-rescue-225efda374c043e3886d349ef724c79e.img
done
[anaconda root@localhost /]# grub2-install /dev/sda
Installing for i386-pc platform.
File descriptor 4 (/) leaked on vgs invocation. Parent PID 29866: grub2-install
File descriptor 4 (/) leaked on vgs invocation. Parent PID 29866: grub2-install
File descriptor 4 (/) leaked on vgs invocation. Parent PID 29866: grub2-install
File descriptor 7 (/) leaked on vgs invocation. Parent PID 29866: grub2-install
File descriptor 8 (/) leaked on vgs invocation. Parent PID 29866: grub2-install
Installation finished. No error reported.
[anaconda root@localhost /]# dracut --kver 4.0.4-301.fc22.x86_64 --force

Now we can exit the chroot, unmount all filesystems and reboot into our new system:

[anaconda root@localhost /]# exit
exit
[anaconda root@localhost ~]# umount /mnt/sysimage/{dev,run,sys,proc}
[anaconda root@localhost ~]# umount /mnt/sysimage/
[anaconda root@localhost ~]# reboot

To Be Continued

So we have set up the system to have a single BTRFS filesystem (no subvolumes) on top of LVM on top of LUKS and with a custom GRUB that respects the configured default subvolume on the BTRFS filesystem. Here is what an lsblk shows:

[root@localhost ~]# lsblk -o NAME,TYPE,FSTYPE,MOUNTPOINT /dev/sda
NAME                TYPE  FSTYPE      MOUNTPOINT
sda                 disk
`-sda1              part  crypto_LUKS
  `-cryptodisk      crypt LVM2_member
    |-vgroot-lvswap lvm   swap        [SWAP]
    `-vgroot-lvroot lvm   btrfs       /

In a later post I will configure snapper on this system and show how rollbacks can be used to simply revert changes that have been made.

Dusty

Manual Linux Installs with Funky Storage Configurations

Introduction


I often find that my tastes for hard drive configurations on my installed systems is a bit outside of the norm. I like playing with thin LVs, BTRFS snapshots, or whatever new thing there is around the corner. The Anaconda UI has been adding support for these fringe cases but I still find it hard to get Anaconda to do what I want in certain cases.

An example of this happened most recently when I went to reformat and install Fedora 20 on my laptop. Ultimately what I wanted was encrypted root and swap devices and btrfs filesystems on root and boot. One other requirement was that I needed to leave sda4 (a Windows Partition) completely intact. At the end the configuration should look something like:

[root@lintop ~]# lsblk /dev/sda NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 465.8G 0 disk ├─sda1 8:1 0 1G 0 part /boot ├─sda2 8:2 0 4G 0 part │ └─cryptoswap 253:1 0 4G 0 crypt [SWAP] ├─sda3 8:3 0 299.2G 0 part │ └─cryptoroot 253:0 0 299.2G 0 crypt / └─sda4 8:4 0 161.6G 0 part

After a few failed attempts with Anaconda I decided to do a custom install instead.

Custom Install


I used the Fedora 20 install DVD (and thus the Anaconda environment) to do the install but I performed all the steps manually by switching to a different terminal with a prompt. First off I used fdisk to format the disk the way I wanted. The results looked like:
[anaconda root@localhost ~]# fdisk -l /dev/sda Disk /dev/sda: 465.8 GiB, 500107862016 bytes, 976773168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: dos Disk identifier: 0xcfe1cf72 Device Boot Start End Blocks Id System /dev/sda1 * 2048 2099199 1048576 83 Linux /dev/sda2 2099200 10487807 4194304 82 Linux swap / Solaris /dev/sda3 10487808 637945855 313729024 83 Linux /dev/sda4 637945856 976773119 169413632 7 HPFS/NTFS/exFAT

Next I set up the encrypted root (/dev/sda3) device and created a btrfs filesystems on both boot (/dev/sda2) and the encrypted root device (/dev/mapper/cryptoroot):

[anaconda root@localhost ~]# cryptsetup luksFormat /dev/sda3 ... [anaconda root@localhost ~]# cryptsetup luksOpen /dev/sda3 cryptoroot Enter passphrase for /dev/sda3: [anaconda root@localhost ~]# [anaconda root@localhost ~]# mkfs.btrfs --force --label=root /dev/mapper/cryptoroot ... fs created label root on /dev/mapper/cryptoroot ... [anaconda root@localhost ~]# mkfs.btrfs --force --label=boot --mixed /dev/sda1 ... fs created label boot on /dev/sda1 ...

Next, if you want to use the yum cli then you need to install it because some of the files are left out of the environment by default. I show the error you get below and then how to fix it:

[anaconda root@localhost ~]# yum list Traceback (most recent call last): File "/bin/yum", line 28, in import yummain ImportError: No module named yummain [anaconda root@localhost ~]# rpm -ivh --nodeps /run/install/repo/Packages/y/yum-3.4.3-106.fc20.noarch.rpm ...

I needed to set up a repo that used the DVD as the source:

[anaconda root@localhost ~]# cat <<EOF > /etc/yum.repos.d/repo.repo [dvd] name=dvd baseurl=file:///run/install/repo enabled=1 gpgcheck=0 EOF

Now I could mount my root device on /mnt/sysimage and then lay down the basic filesystem tree by installing the filesystem package into it:

[anaconda root@localhost ~]# mount /dev/mapper/cryptoroot /mnt/sysimage/ [anaconda root@localhost ~]# yum install -y --installroot=/mnt/sysimage filesystem ... Complete!

Now I can mount boot and other filesystems into the /mnt/sysimage tree:

[anaconda root@localhost ~]# mount /dev/sda1 /mnt/sysimage/boot/ [anaconda root@localhost ~]# mount -v -o bind /dev /mnt/sysimage/dev/ mount: /dev bound on /mnt/sysimage/dev. [anaconda root@localhost ~]# mount -v -o bind /run /mnt/sysimage/run/ mount: /run bound on /mnt/sysimage/run. [anaconda root@localhost ~]# mount -v -t proc proc /mnt/sysimage/proc/ mount: proc mounted on /mnt/sysimage/proc. [anaconda root@localhost ~]# mount -v -t sysfs sys /mnt/sysimage/sys/ mount: sys mounted on /mnt/sysimage/sys.

Now ready for the actual install. For this install I just went with a small set of packages (I'll use yum once the system is up to add on what I want later).

[anaconda root@localhost ~]# yum install -y --installroot=/mnt/sysimage @core @standard kernel grub2 grub2-tools btrfs-progs ... Complete!

After the install there are a few housekeeping items to take care of. I started with populating crypttab, populating fstab, changing the root password, and touching /.autorelabel to trigger an selinux relabel on first boot:

[anaconda root@localhost ~]# chroot /mnt/sysimage/ [anaconda root@localhost /]# cat <<EOF > /etc/crypttab cryptoswap /dev/sda2 /dev/urandom swap cryptoroot /dev/sda3 - EOF [anaconda root@localhost /]# cat <<EOF > /etc/fstab LABEL=boot /boot btrfs defaults 1 2 /dev/mapper/cryptoswap swap swap defaults 0 0 /dev/mapper/cryptoroot / btrfs defaults 1 1 EOF [anaconda root@localhost /]# passwd --stdin root <<< "password" Changing password for user root. passwd: all authentication tokens updated successfully. [anaconda root@localhost /]# touch /.autorelabel

Next I needed to install grub and make a config file. I set the grub kernel command line arguments and then generated a config. The config needed some fixing up (I am not using EFI on my system, but grub2-mkconfig thought I was because I had booted through EFI off of the install CD).

[anaconda root@localhost /]# echo 'GRUB_CMDLINE_LINUX="ro root=/dev/mapper/cryptoroot"' > /etc/default/grub [anaconda root@localhost /]# grub2-mkconfig -o /boot/grub2/grub.cfg Generating grub.cfg ... Found linux image: /boot/vmlinuz-3.11.10-301.fc20.x86_64 Found initrd image: /boot/initramfs-3.11.10-301.fc20.x86_64.img Found linux image: /boot/vmlinuz-0-rescue-81c04e9030594ef6a5265a95f58ccf98 Found initrd image: /boot/initramfs-0-rescue-81c04e9030594ef6a5265a95f58ccf98.img done [anaconda root@localhost /]# sed -i s/linuxefi/linux/ /boot/grub2/grub.cfg [anaconda root@localhost /]# sed -i s/initrdefi/initrd/ /boot/grub2/grub.cfg [anaconda root@localhost /]# grub2-install -d /usr/lib/grub/i386-pc/ /dev/sda Installation finished. No error reported.

NOTE: grub2-mkconfig didn't find my windows partition until I rebooted into the system and ran it again.

Finally I re-executed dracut to pick up the crypttab, exited the chroot, unmounted the filesystems, and rebooted into my new system:
[anaconda root@localhost /]# dracut --kver 3.11.10-301.fc20.x86_64 --force [anaconda root@localhost /]# exit [anaconda root@localhost ~]# umount /mnt/sysimage/{boot,dev,run,sys,proc} [anaconda root@localhost ~]# reboot

After booting into Fedora I was then able to run grub2-mkconfig again and get it to recognize my (untouched) Windows partition:

[root@localhost /]# grub2-mkconfig -o /boot/grub2/grub.cfg Generating grub.cfg ... Found linux image: /boot/vmlinuz-3.11.10-301.fc20.x86_64 Found initrd image: /boot/initramfs-3.11.10-301.fc20.x86_64.img Found linux image: /boot/vmlinuz-0-rescue-375c7019484a45838666c572d241249a Found initrd image: /boot/initramfs-0-rescue-375c7019484a45838666c572d241249a.img Found Windows 7 (loader) on /dev/sda4 done

And that's pretty much it. Using this method you can have virtually any hard drive setup that you desire. Hope someone else can find this useful.

Dusty

P.S. You can start sshd in anaconda by running systemctl start anaconda-sshd.service.

Booting Anaconda from Software RAID1 Device


Sometimes you just want to boot Anaconda from a software raid device that houses both the stage1 (initrd.img) and stage2 (install.img) images. There are various reasons to do this some of which include:
  • Booting Anaconda into rescue mode from hard drive (RAID 1)
  • Installing directly from hard drive (RAID 1)
  • Running PreUpgrade (Now Deprecated)
Running Anaconda from a RAID 1 device is unsupported at least up until the rhel 6.4 version of Anaconda and is documented in BZ #500004 (note that this is a "feature", not a bug) . It may be supported now with the new Anaconda redesign , but I havn't put in the time to find out yet.

That being said, a good workaround mentioned in the comments of the bug report is to simply attach an install cd that contains the same version of Anaconda you are trying to boot and it will automatically find it and continue on. However, if you often need to boot Anaconda like this it can be tedious and there is another way that may be more desireable to some.

Since /boot/ is set up on a software RAID 1 each member device contains an exact copy of what is on the RAID device. This means that you can directly mount any of the member devices as long as you specify the filesystem type. This is exactly where Anaconda has a problem.

The kernel and stage1 are loaded by grub, which sucessfully ignores the fact that the device is a part of a raid and just treats it like an ext filesystem. Anaconda on the other hand attempts to mount the specified device to find the stage2 image. In doing this Anaconda calls /bin/mount and specifies an fs type of auto ( -t auto ). Since the device has an MD superblock, mount fails to detect it is ext4 and does not mount the device.

What is the solution for this?? Well, we need to get rid of the superblock :)

As a brief example I will show how to set up Anaconda to boot into rescue mode from a software RAID 1. First we need to copy the kernel, stage1, and stage2 images from an install cd and into the /boot/ filesystem.

[root@cent64 ~]# mkdir /mnt/dusty [root@cent64 ~]# mount /dev/sr0 /mnt/dusty/ mount: block device /dev/sr0 is write-protected, mounting read-only [root@cent64 ~]# [root@cent64 ~]# mkdir /boot/images [root@cent64 ~]# cp /mnt/dusty/images/install.img /boot/images/ [root@cent64 ~]# cp /mnt/dusty/isolinux/initrd.img /boot/ [root@cent64 ~]# cp /mnt/dusty/isolinux/vmlinuz /boot/

Next, add a new entry into the grub.conf so that we can choose to boot into the Anaconda rescue environment. The new entry is shown below. Note that I am specifying where to find the stage2 image from the kernel command line.

title rescue root (hd0,0) kernel /vmlinuz rescue stage2=hd:sda1:/ initrd /initrd.img

Now we must zero out the superblock on /dev/sda1 which will enable Anaconda's mount to auto detect the ext4 fs on sda1.

[root@cent64 ~]# umount /boot [root@cent64 ~]# mdadm --stop /dev/md0 mdadm: stopped /dev/md0 [root@cent64 ~]# mdadm --zero-superblock /dev/sda1 [root@cent64 ~]# [root@cent64 ~]# mdadm --examine /dev/sda1 mdadm: No md superblock detected on /dev/sda1.

After rebooting and manually selecting "rescue" at the grub screen I am able to successfully boot into rescue mode using Anaconda!!!

Taking a peak at the program.log from Anaconda shows me exactly what command was run to mount the device.

Running... /bin/mount -n -t auto -o ro /dev/sda1 /mnt/isodir

Now we achieved what we wanted but the fun isn't all over.. After doing this you must fix the superblock on sda1. After boot you can see that sda1 is not being mirrored.

[root@cent64 ~]# cat /proc/mdstat Personalities : [raid1] md0 : active raid1 sdb1[1] 409536 blocks super 1.0 [2/1] [_U] md1 : active raid1 sda2[0] sdb2[1] 20544384 blocks super 1.1 [2/2] [UU] bitmap: 0/1 pages [0KB], 65536KB chunk unused devices:

You can add it back into the array like so. Once it is done syncing it is good as new again:

[root@cent64 ~]# mdadm --add /dev/md0 /dev/sda1 mdadm: added /dev/sda1 [root@cent64 ~]# [root@cent64 ~]# cat /proc/mdstat Personalities : [raid1] md0 : active raid1 sda1[2] sdb1[1] 409536 blocks super 1.0 [2/1] [_U] [==>..................] recovery = 11.0% (45248/409536) finish=0.2min speed=22624K/sec md1 : active raid1 sda2[0] sdb2[1] 20544384 blocks super 1.1 [2/2] [UU] bitmap: 1/1 pages [4KB], 65536KB chunk [root@cent64 ~]# [root@cent64 ~]# mdadm --examine /dev/sda1 /dev/sda1: Magic : a92b4efc Version : 1.0 Feature Map : 0x0 Array UUID : d23c9678:2918b450:86f94f62:1a38d114 Name : cent64:0 (local to host cent64) Creation Time : Sun May 5 14:40:41 2013 Raid Level : raid1 Raid Devices : 2 Avail Dev Size : 819176 (400.06 MiB 419.42 MB) Array Size : 409536 (400.00 MiB 419.36 MB) Used Dev Size : 819072 (400.00 MiB 419.36 MB) Super Offset : 819184 sectors State : clean Device UUID : 805e3791:be06e475:619ee8c0:4e2599f4 Update Time : Sun May 5 15:31:00 2013 Checksum : edd172ef - correct Events : 19 Device Role : Active device 0 Array State : AA ('A' == active, '.' == missing) [root@cent64 ~]#


NOTE: As a quick note before I part I would like to stress the fact that the steps taken in the example above may not satisfy all needs. Please understand what you are doing before you do it. As in all cases, backups are your friend.

Until Next Time..

Dusty