A Random Walk Down Tech Street

PoC: Leveraging Butane during Ignition boot

Tue, 12 Aug 2025 00:00:00 +0000

Converting Butane to Ignition on Boot

We’ve received anecdotal evidence over the years that the transpilation step where you must convert a Butane YAML configuration into Ignition JSON before staring your CoreOS instance is a pain for users to understand and implement. CoreOS developers also feel this pain at times.

What if you could feed either a Butane YAML config or an Ignition JSON config to your instance and it would just do the right thing regardless?

Transferring files over a serial connection

Sat, 03 May 2025 00:00:00 +0000

Problem

Sometimes you just don’t have a stable (or even functioning network) to leverage when dealing with a piece of hardware. In all my years in Linux admin/development I’ve always worked around this problem with some sort of sneakernet, but there is another way.

Typically when bringing up new hardware one of the most basic interfaces you have is a serial console connection and I just learned recently how to transfer files over that serial console connection rather than requiring a net (tcp/ip) connection.

Fedora BTRFS+Snapper - The Fedora 41 Edition

Tue, 07 Jan 2025 00:00:00 +0000

History

It’s a new year and it has been 5 years since the last time I updated this blog post series. This time I’m updating to Fedora 41 and also bringing forward a lot of changes I’ve made over the last 5 years.

As mentioned previously for now I’m using Fedora Silverblue on my laptop systems and continuing with a BTRFS+snapper setup for my desktop. This series describes the BTRFS+snapper setup.

In the past I have documented this setup and all the steps I took in detail for Fedora 22 (part1 and part2), 24, 25, 27, 29 and 31. This is a condensed continuation of those posts for Fedora 41, but there are a few changes I’ve made.

Configuring and using iSCSI

Fri, 10 May 2024 00:00:00 +0000

Recently I looked into enabling and testing multipath on top of iSCSI for Fedora and Red Hat CoreOS. As part of that process I had the opportunity to learn about iSCSI, which I had never played with before. I’d like to document for my future self how to go about setting up an iSCSI server and how to then access the exported devices from another system.

Setting up an iSCSI server

First off there are a few good references that were useful when setting this up. The RHEL 9 documentation for managing storage devices was one. The other was the targetcli man page.

Using virtiofs with libvirt/virt-install

Fri, 08 Sep 2023 00:00:00 +0000

Recently we switched our 9p filesystem usage in CoreOS Assembler to use virtiofs.

This is the technology behind a lot of new lightweight container VM technology like kata-containers and libkrun, but can also be easily used with libvirt.

Running as non-root using qemu:///session

Currently the virtiofs integration doesn’t work as non-root via a qemu:///session connection. There is an oustanding RFE for this upstream and downstream in RHEL that can be followed for updates.

virtiosfs with virt-install

I typically use virt-install to automate creation of my libvirt virtual machines. Using virt-install we can use the --filesystem= and --memorybacking= parameters to get what we want. The --memorybacking= parameter will add shared memory to the instance, which is required for virtiofs.

Columbia University and Fedora CoreOS

Tue, 01 Aug 2023 00:00:00 +0000

Earlier this summer I spoke at the Fedora 38 release party about Fedora CoreOS. As part of this I had the privilege of giving the presentation alongside Marc Pusey from Columbia University. Marc talked about how Columbia University uses Fedora CoreOS and his experience participating in the Fedora and Fedora CoreOS communities.

It was a real treat and quite nice to see Fedora CoreOS being applied in such diverse ways for research and learning.

NetworkManager: Limiting Bond Subordinate devices by MAC Address

Thu, 16 Mar 2023 00:00:00 +0000

Someone recently asked me about locking down a bond to specific NIC devices within the machine. Specifically they were concerned with the sometimes unpredictable nature of NIC naming in Linux.

While there has been a lot of effort to make NIC naming more predictable, it turns out with the networking configuration stack we are using in Fedora/RHEL (NetworkManager) you don’t even really need to care about the NIC device names if you know the MAC Addresses of the interfaces you want to use.

Running FCOS on your Raspberry Pi 4

Tue, 02 Nov 2021 00:00:00 +0000

Note: A more permanent version of this tutorial exists in the Fedora CoreOS documentation.

Fedora CoreOS recently started producing 64-bit ARM (aarch64) artifacts. These images can be used as the Operating System for the Raspberry Pi 4 device. Before trying to get FCOS up and running on your Raspberry Pi4 you’ll want to Update the EEPROM to the latest version and choose how you want to boot the Raspberry Pi 4. There are two options:

CoreOS install via Live ISO --copy-network

Wed, 18 Nov 2020 00:00:00 +0000

A couple of us recently gave an update to our Customer Experience team at Red Hat on the improvements that were made in Red Hat CoreOS for OpenShift 4.6. My part of the presentation focused on the new Live ISO that is now used for Fedora/Red Hat CoreOS installations and also the improvements that we made for being able to copy the install environment networking configuration into the installed system via coreos-installer --copy-network.

GCP Quickstart Guide for OpenShift OKD

Wed, 07 Oct 2020 00:00:00 +0000

Introduction

I recently did a blog post series. showing how to get started with OpenShift OKD on Fedora CoreOS for DigitalOcean. For that series I wrote a script to do most of the heavy lifting because DigitalOcean isn’t a native supported platform by the OpenShift installer.

Today I’ll show off how to get started in GCP, which is supported natively by the OpenShift installer. This makes it much easier to get started because most of the heavy lifting (including infrastructure bringup) is done by the installer itself.

OpenShift OKD on Fedora CoreOS on DigitalOcean Part 4: Recorded Demo

Mon, 28 Sep 2020 00:00:00 +0000

NOTE: The fourth post of this series is available here.

This blog post is the fifth in a series that illustrates how to set up an OpenShift OKD cluster on DigitalOcean.

Back on August 17th I highlighted this blog post series with a presentation and demo for the OKD working group’s Demo Marathon.

The video is posted on YouTube and also available below. If you’re interested take some time and watch the whole process work and see a cluster up and running at the end.

OpenShift OKD on Fedora CoreOS on DigitalOcean Part 3: Upgrading

Sun, 27 Sep 2020 00:00:00 +0000

Introduction

NOTE: The third post of this series is available here.

This blog post is the fourth in a series that illustrates how to set up an OpenShift OKD cluster on DigitalOcean. The third post in the series covered further configuration of a cluster once it’s already up and running. At this point you should have a cluster up and running and configured with custom TLS certificates and user login’s outsourced to some other identity management service.

OpenShift OKD on Fedora CoreOS on DigitalOcean Part 2: Configuration

Sun, 23 Aug 2020 00:00:00 +0000

Introduction

NOTE: The second post of this series is available here.

This blog post is the third in a series that illustrates how to set up an OpenShift OKD cluster on DigitalOcean. The second post in the series covered the automated deployment and teardown of a cluster using the digitalocean-okd-install script. At this point you should have a cluster up and running and ready to be further customized.

Set Up Custom TLS Certificates

In the first post in this series we mentioned that you may want to have valid certificates for your cluster. For this example we used certbot to talk with Let's Encrypt to get us some certificates for use with our cluster. Assuming the files are in the letsencrypt/ directory we can add the certificates to our cluster like so:

OpenShift OKD on Fedora CoreOS on DigitalOcean Part 1: Deployment

Thu, 13 Aug 2020 00:00:00 +0000

Introduction

NOTE: The first post of this series is available here.

This blog post is the second in a series that illustrates how to set up an OpenShift OKD cluster on DigitalOcean. The first post in the series covered some background information and pre-requisites needed for deploying a cluster. At this point you should have chosen the domain for your cluster, set up your registrar to point to DigitalOcean nameservers, installed all necessary software (doctl, openshift-install, oc, aws cli, etc..), and configured appropriate credentials in your environment (DIGITALOCEAN_ACCESS_TOKEN, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY).

OpenShift OKD on Fedora CoreOS on DigitalOcean Part 0: Preparation

Tue, 28 Jul 2020 00:00:00 +0000

Introduction

This blog post is the first in a series that will illustrate how to set up an OpenShift OKD cluster on DigitalOcean using the bare metal install documentation (user provisioned infrastructure). OKD has tight integrations with the Operating System and uses Fedora CoreOS as a platform for driving the underlying infrastructure, thus we’ll be deploying on top of Fedora CoreOS images inside of DigitalOcean.

The documentation for OKD is pretty comprehensive, but there is nothing like having a guide to help fill in some of the gaps and show an example of it working with real world values. This series aims to do just that.

The K9s TUI for Kubernetes

Sat, 18 Jul 2020 00:00:00 +0000

If you’ve ever had a chance to “nerd out” with me much I’ve probably told you about at least one TUI that I use to make my daily work life easier. Some of my favorites include htop, tig, tmux (not sure if tmux counts), etc..

Lately I’ve been finding myself using Kubernetes/OpenShift much more and I have often thought to myself I wish a nice TUI existed to get in and navigate resources in a Kubernetes environment. The various web interfaces are nice and comprehensive, and the CLI is awesome too, but each of them are too far on the fringes of what I’ll define as my sweet spot for discoverability vs speed vs power. A well designed TUI sits right in the middle and makes me feel right at home.

Automating a Custom Install of Fedora CoreOS

Sat, 04 Apr 2020 00:00:00 +0000

Introduction

With Fedora CoreOS we currently have two ways to do a bare metal install and get our disk image onto the spinning rust of a “non-cloud” server. You can use coreos.inst* kernel arguments to automate the install, or you can boot the Live ISO and get a bash prompt where you can then run coreos-installer directly after doing whatever hardware/network discovery that is necessary. This means you either do a simple automated install where you provide all of the information up front or you are stuck doing something interactive. However, because we use a Live ISO that boots full Fedora CoreOS there is a third option.

Network teaming using NetworkManager keyfiles on Fedora CoreOS

Thu, 05 Mar 2020 00:00:00 +0000

Introduction

NetworkManager allows connections to be defined in a configuration file known as a keyfile , which is a simple .ini-style formatted file with different key=value pairs. In Fedora CoreOS we’ve elected to use NetworkManager with keyfiles as the way to configure networking. In case you have a standard networking environment with NICs requesting DHCP then you probably won’t need to configure networking. However, if you’d like to have a static networking config or if you’d like to do something more complicated (like configure network teaming for a few interfaces) then you’ll need to create a keyfile that NetworkManager will then use to configure the interfaces on the machine.

virt-install: boot from specific kernel/initrd just for install

Thu, 30 Jan 2020 00:00:00 +0000

Introduction

For some time now with virt-install (developed under virt-manager) you have been able to specify a kernel and initial ramdisk to start a VM with. The only problem is that the VM will always start with that kernel/initrd (unless you change the definition manually). If you are rapidly testing operating system installations this can be problematic.

On the one hand, providing the kernel/initrd allows one to automate the install process from a Linux terminal, or even a script, by specifying the kernel/initrd and also the kernel command line options. However, it only gives us half the picture, because you’d then have to hand edit the libvirt definition of the machine to see if the installed machine was viable, OR you’d be lazy and just throw away the installed machine and assume it was good because the installation process finished without error; BAD.

Devconf.cz 2020 Fedora CoreOS Lab

Thu, 23 Jan 2020 00:00:00 +0000

Setting Up For The Lab

This lab uses a Fedora CoreOS image and several utilities (fcct, ignition-validate) to introduce a user to provisioning and exploring a Fedora CoreOS system. This lab is written targeting a Linux environment with a working libvirt/kvm setup.

To perform this lab you need to download the tar archive at this link (signed checksum file) and extract it.

We recommend extracting it into your home directory like so:

Fedora BTRFS+Snapper - The Fedora 31 Edition

Sun, 29 Dec 2019 00:00:00 +0000

History

It’s almost 2020. Fedora 31 came out a month back and I’m just getting around to converting my desktop system to Fedora 31. As mentioned before, for my laptop systems I’ve moved on to Fedora Silverblue. As I continue to containerize my workflows I’m moving more and more of my daily workflows into Flatpaks from the Fedora registry, pet containers (via toolbox) and, single purpose containers. As I continue to convert my workflows into containers I’ll stick with the BTRFS+snapper setup for my desktop system, which still has the benefits of being able to snapshot and rollback the entire system by leveraging BTRFS snapshots, and a tool called snapper.

Fedora Atomic Host Nearing End Of Life

Thu, 21 Nov 2019 00:00:00 +0000

cross posted with this Project Atomic blog post

TL;DR Fedora 29 will be End Of Life soon. With it Fedora Atomic Host will have its last incremental release (based on the Fedora 29 stream). Please move to the Fedora CoreOS preview if you can.

Last year we introduced the plans for Fedora CoreOS including that Fedora CoreOS would be the successor to Fedora Atomic Host and Container Linux (from CoreOS Inc.). As part of that succession plan we decided that Fedora 29 Atomic Host would be the last stream of Fedora Atomic Host to be released.

Running a script on bootup via Ignition

Wed, 06 Nov 2019 00:00:00 +0000

Introduction

With Fedora CoreOS Ignition is being used to configure nodes on first boot. While Ignition json configs are not intended to be a tool that users typically interact with (we are building tooling like fcct for that) I’ll show you an example of how to deliver a script to a Fedora CoreOS (or RHEL CoreOS) host so that it will be run on first boot.

Write the script

Let’s say we have a small script we want to run that updates the issuegen from console-login-helper-messages to output the node’s public IPv4 address on the serial console during bootup.

Update on Easy PXE boot testing post: minus PXELINUX

Fri, 13 Sep 2019 00:00:00 +0000

Introduction

This is an update to my previous post about easily testing PXE booting by using libvirt + iPXE.

Several people have notified me (thanks Lukas Zapletal and others) that instead of leveraging PXELINUX that I could just use an iPXE script to do the same thing. I hadn’t used iPXE much so here’s an update on how to achieve the same goal using an iPXE script instead of a PXELINUX binary+config.

Fedora BTRFS+Snapper - The Fedora 29 Edition

Sun, 06 Jan 2019 00:00:00 +0000

History

It’s 2019 and I’m just getting around to converting my desktop system to Fedora 29. For my work laptop I’ve moved on to Fedora Silverblue (previously known as Atomic Workstation) and will probably move my desktop there soon too as I’ve had a good experience so far. For now I’ll stick my desktop system to this old setup with BTRFS+snapper where I am able to snapshot and rollback the entire system by leveraging BTRFS snapshots, and a tool called snapper.

Easy PXE boot testing with only HTTP using iPXE and libvirt

Fri, 04 Jan 2019 00:00:00 +0000

Update: A future post explains how to do this even easier without PXELINUX.

Introduction

Occasionally I have a need to test out a PXE install workflow. All of this is super easy if you have a permanent PXE infrastructure you maintain which traditionally has consisted of DHCP, TFTP and HTTP/FTP servers. What if I just have my laptop and want to test something in a VM? It turns out it’s pretty easy to do using libvirt and a simple http server.

Fedora 28->29 Atomic Host Upgrade Guide

Wed, 31 Oct 2018 00:00:00 +0000

cross posted with this Project Atomic blog post

Introduction

This week we put out the first release of Fedora 29 Atomic Host. This will be the last major release of Fedora Atomic Host as we prepare for Fedora CoreOS which will be released in Fedora 30.

In this post we’ll quickly list some known issues and then talk about updating an existing Fedora 28 Atomic Host system to Fedora 29. We’ll cover preparing the system for upgrade and performing the upgrade.

RPM-OSTree Bisecting Helps Track Down Boot Timeout Issue

Sat, 20 Oct 2018 00:00:00 +0000

Introduction

Last time I talked about using rpm-ostree-bisect, a tool that I wrote to automatically bisect the history of an OSTree remote in order to find the exact commit when a problem was introduced. I recently put the tool to the test again.

The Problem

Recently a user reported an issue where their system was seeing timeouts on boot. They determined that if they removed the resume=/dev/mapper/fedora-swap argument from the kernel command line then the system would boot without timing out on the swap device (i.e. an extra 90 seconds added to boot time).

Automated Bisect Testing Of An Entire OS with RPM-OSTree

Sun, 10 Jun 2018 00:00:00 +0000

Introduction

Occasionally in OS land we’ll come across a bug that snuck its way into a build and has been in the wild for a while before anyone notices it. One example is a recent bug (originally discovered by the community of CoreOS Container Linux) where the jumbo packet MTU size of 9001 was no longer getting set properly on EC2 instances.

So we have this bug, and we know things used to work. I fired up the first and last releases of F28 Atomic Host. Both had the problem. I then went all the way back to the first release of F27 Atomic Host and fired up an AMI from that release. On that release the MTU looks good at 9001:

Pinning Deployments in OSTree Based Systems

Tue, 22 May 2018 00:00:00 +0000

cross posted with this Project Atomic blog post

Introduction

RPM-OSTree/OSTree conveniently allows you to rollback if you upgrade and don’t like the upgraded software. This is done by keeping around the old deployment; the old software you booted in to. After a single upgrade you’ll have a booted deployement and the rollback deployment. On the next upgrade the current rollback deployment will be discarded and the current booted deployment will become the new rollback deployment.

April Fedora Infrastructure Hackfest

Mon, 23 Apr 2018 00:00:00 +0000

Introduction

Earlier this month I was lucky enough to attend the 2018 Fedora Infrastructure Hackfest. It’s always a treat to hang out with some of the people who really make Fedora tick. Sinny Kumari and I were there to help represent the Atomic Working Group, and also get some face time with each other to learn and hack on a few things related to the Atomic Working group.

The Hackfest was held in Paul Frield’s hometown of Fredricksburg, VA. Since I live in Raleigh, NC I decided it would be nice to take the train since I don’t often get to take the train in the southeast. As can be expected the train was a little late, but got us there without a problem and was a pretty good experience.

Pruning Policy for Specific Branches of OSTree Repos

Sun, 04 Mar 2018 00:00:00 +0000

Introduction

In Fedora we are moving to a unified OSTree repo structure where there is a single OSTree repository that is the remote for all branches of Fedora (rawhide, branched, stable, etc). As part of this we want to be able to define different retention policies for different branches within the repository. For rawhide we’ll retain a few weeks worth of commits, but for stable we don’t want to prune any of the commits.

Firewalld in Atomic Host

Thu, 01 Feb 2018 00:00:00 +0000

cross posted with this Project Atomic blog post

TL;DR

Fedora Atomic Host (and derivatives) will now include the firewalld package in the base OSTree that is tested, delivered, and released every two weeks. Existing users should observe no change as it won’t be enabled by default.

Firewalld in Atomic Host

In the past we have had requests to have firewalld in Atomic Host to enable a better interface into firewall management for administrators and management software. It turns out that if you have lots of rules to manage, or even multiple pieces of software trying to manage different sets of rules on a single system, then iptables becomes a limitation pretty quickly.

Fedora BTRFS+Snapper - The Fedora 27 Edition

Sun, 17 Dec 2017 00:00:00 +0000

History

I’m back again with the Fedora 27 edition of my Fedora BTRFS+Snapper series. As you know, in the past I have configured my computers to be able to snapshot and rollback the entire system by leveraging BTRFS snapshots, a tool called snapper, and a patched version of Fedora’s grub2 package. I have some great news this time! You no longer need a patched version of Fedora’s grub package in order to pull this off. Recently Fedora developer Peter Jones, Fedora contributor Neal Gompa and I got together and managed to get these patches into Fedora’s grub.

Fedora 26->27 Atomic Host Upgrade Guide

Tue, 14 Nov 2017 00:00:00 +0000

cross posted with this Project Atomic blog post

Introduction

This week we put out the first release of Fedora 27 Atomic Host. Some quick notes:

In Fedora 27 Atomic Host we removed kubernetes from the base OSTree. See Appendix A: Upgrading Systems with Kubernetes for more information.
For Fedora 27 we are currently sticking with the non-unified repo approach as opposed to a unified repo. TL;DR nothing is changing for now but we expect to implement a unified repo as described here during the F27 release cycle.

Setting up an Atomic Host Build Server

Thu, 05 Oct 2017 00:00:00 +0000

Introduction

Hosting your own Atomic Host OSTree can be useful from time to time. Maybe you want to try out something new or maybe you want to permanently build your own custom tree and use it forever. It can be quite easy to set up a build server and host the contents, especially for personal use.

This post will walk through setting up a server to do builds and also hosting the content over http.

Atomic Host 101 Lab Part 5: Containerized and Non-Containerized Applications

Sun, 03 Sep 2017 00:00:00 +0000

Introduction

In Part 4 of this series we learned about package layering and experimental features of atomic host OSTree mutations. This included installing packages from external repositories as well as removing and replacing components of the base OSTree that was delivered with Atomic Host. We also converted our localweb service to be hosted by a local docker container running the httpd software rather than Python 3.

In this section of the lab we’ll talk a litte bit more about Containerized and Non-Containerized Applications and the role Atomic Host plays.

Atomic Host 101 Lab Part 4: Package Layering, Experimental Features

Sat, 02 Sep 2017 00:00:00 +0000

Introduction

In Part 3 of this series we learned about rebasing, upgrading, and performing rollbacks on Atomic Host. We also learned how files are restored during a rollback operation and how to inspect the differences in RPM content between each commit in the OSTree history of an Atomic Host using the rpm-ostree command line tool. In this section we will cover the following topics from the outline in Part 0.

Package Layering
Experimental Features (livefs, remove, replace)

Adding Packages to Atomic Host via Package Layering

When Atomic Host was first released we could not change much about the the delivered software on the system. Over time we developed a system for layering packages on top of what was provided by the base OSTree to allow the flexibility needed for those few packages that, for whatever reason, we may not want to put into a container.

Atomic Host 101 Lab Part 3: Rebase, Upgrade, Rollback

Fri, 01 Sep 2017 00:00:00 +0000

Introduction

In Part 2 of this series we learned about configuring container storage on Atomic Host. In this section we will cover the following topics from the outline in Part 0.

Atomic Host Rebasing
Atomic Host Upgrades and Rollbacks
Browsing OS History

Rebasing

One of the more fascinating aspects of Atomic Host techology is that you can rebase to completely different operating system trees. Let’s take this to an extreme and go from the newer technology in Fedora to the older (more stable) technology in CentOS. We’ll achieve by rebasing to an OSTree commit that was built from CentOS 7 RPMs:

Atomic Host 101 Lab Part 2: Container Storage

Thu, 31 Aug 2017 00:00:00 +0000

Introduction

In Part 1 of this series we learned a little about the technology behind Atomic Host and how to interact with a deployed system. In this section we will cover the Configuring Storage for Containers topic from the outline in Part 0.

History of Container Storage

One of the early goals of Atomic Host was to be a good platform for running containerized workloads. This is still a fundamental goal of Atomic Host and certainly includes making sure that the container runtime (currently the docker daemon) has proper storage configuration such that it can get a balance of good performance and stability.

Atomic Host 101 Lab Part 1: Getting Familiar

Wed, 30 Aug 2017 00:00:00 +0000

Introduction

In Part 0 of this series we helped get a Fedora 26 Atomic Host system set up for the rest of this lab. In this section we will cover the following topics from the outline:

Getting Familiar With Atomic Host
Viewing Changes To A Deployed System

Getting Familiar

Atomic Host is built on top of underlying technology known as OSTree and leveraged by an RPM aware higher level technology known as rpm-ostree. rpm-ostree is able to build and deliver OSTrees built out of RPMs. Once built, an OSTree commit can be installed to a server just like a traditional OS. New OSTree commits are created by a build system and a server can pull down and apply updates, similar to a git pull.

Atomic Host 101 Lab Part 0: Preparation

Tue, 29 Aug 2017 00:00:00 +0000

Introduction

While Atomic Host has been around since 2014 there are still a lot of people that aren’t as familiar with the technology. The Atomic team within Red Hat, along with numerous other upstream contributors, have brought the OSTree and RPM-OSTree technology a long way. At the Fedora user and contributor conference (known as Flock) this week we will be giving a lab on Atomic Host designed to let new users learn about Atomic Host. The audience for this lab is anyone familiar with Linux and interested in learning a new technology.

Fedora 26 Atomic Host August 08 Release

Wed, 09 Aug 2017 00:00:00 +0000

cross posted with this Project Atomic blog post

A new Fedora Atomic Host update is available via an OSTree commit:

Commit: f6331bcd14577e0ee43db3ba5a44e0f63f74a86e3955604c20542df0b7ad8ad6
Version: 26.101

In this release we have fixed an issue with our qcow and vagrant images from the 20170723 release. If you used the qcow or vagrant images from that release then please make sure you are following the fedora/26/x86_64/atomic-host ref. See this Atomic Working Group issue for more details.

The diff between this and the previous released version is:

How Do We Create OSTree Repos and Artifacts in Fedora

Tue, 08 Aug 2017 00:00:00 +0000

a more permanent version of this content lives here

Introduction

NOTE: For background on OSTree check out the docs.

When you want to create a new OSTree using rpm-ostree you usually define a few yum repos, and then a json file that says what rpms you want to be composed in the tree. You then run an rpm-ostree compose tree command to create the commit in the ostree repo. Once the ostree commit has been created you can then create installer images (ISOs) and cloud/VM images (qcow, etc) from that ostree.

Fedora 25->26 Atomic Host Upgrade Guide

Thu, 03 Aug 2017 00:00:00 +0000

cross posted with this Project Atomic blog post and this Fedora Magazine post

Introduction

In July we put out the first and second releases of Fedora 26 Atomic Host. In this blog post we’ll cover updating an existing Fedora 25 Atomic Host system to Fedora 26. We’ll cover preparing the system for upgrade and performing the upgrade.

NOTE: If you really don’t want to upgrade to Fedora 26 see the later section: Appendix B: Fedora 25 Atomic Host Life Support.

F26 Atomic/Cloud Test Day June 20th!

Thu, 15 Jun 2017 00:00:00 +0000

cross posted from this_ fedora magazine post

Now that the Fedora Beta has been officially released the Fedora Atomic Working Group and Fedora Cloud SIG would like to get the community together next week to find and squash some bugs. We are organizing a test day for Tuesday, June 20th.

For this event we'll test both Atomic Host content and Fedora Cloud Base content. Vagrant Boxes will be available to test with as well. See the Fedora Atomic Host Pre-Release Page for links to artifacts for Fedora Atomic Host and the Alternative Downloads Beta Page for links for the Beta Cloud Base Images. We have qcow, AMI, and ISO images ready for testing.

Fedora Atomic Host Available in DigitalOcean

Thu, 27 Apr 2017 00:00:00 +0000

Cross posted with this_ Project Atomic Blog post

---

With the latest release of Fedora Atomic Host we are now live in DigitalOcean! This was a popular user request and thanks to the folks at DigitalOcean and the Fedora Atomic Working Group we now have Fedora Atomic Host as an option when creating a droplet. Go ahead and spin up a droplet in the web interface or via the doctl CLI today!

Matching Fedora OSTree Released Content With Each 2 Week Atomic Release

Tue, 14 Feb 2017 00:00:00 +0000

Cross posted with this_ Project Atomic Blog post

TL;DR: The default Fedora cadence for updates in the RPM streams is once a day. Until now, the OSTree-based updates cadence has matched this, but we're changing the default OSTree update stream to match the Fedora Atomic Host image release cadence (once every two weeks).

---

Fedora BTRFS+Snapper - The Fedora 25 Edition

Sun, 12 Feb 2017 00:00:00 +0000

History

I'm back again with the Fedora 25 edition of my Fedora BTRFS+Snapper series. As you know, in the past I have configured my computers to be able to snapshot and rollback the entire system by leveraging BTRFS snapshots, a tool called snapper, and a patched version of Fedora's grub2 package. I have updated the patchset (patches taken from SUSE) for Fedora 25's version of grub and the results are available in this git repo.

Installing an OpenShift Origin Cluster on Fedora 25 Atomic Host: Part 2

Mon, 12 Dec 2016 00:00:00 +0000

Cross posted with this_ Project Atomic Blog post

Introduction

In part 1 of this series we used the OpenShift Ansible Installer to install Openshift Origin on three servers that were running Fedora 25 Atomic Host. The three machines we'll be using have the following roles and IP address configurations:

+-------------+----------------+--------------+
|     Role    |   Public IPv4  | Private IPv4 |
+=============+================+==============+
| master,etcd | 54.175.0.44    | 10.0.173.101 |
+-------------+----------------+--------------+
|    worker   | 52.91.115.81   | 10.0.156.20  |
+-------------+----------------+--------------+
|    worker   | 54.204.208.138 | 10.0.251.101 |
+-------------+----------------+--------------+

In this blog, we'll explore the installed Origin cluster and then launch an application to see if everything works.

Installing an OpenShift Origin Cluster on Fedora 25 Atomic Host: Part 1

Wed, 07 Dec 2016 00:00:00 +0000

Cross posted with this_ Project Atomic Blog post

Introduction

Openshift Origin is the upstream project that builds on top of the Kubernetes platform and feeds into the OpenShift Container Platform product that is available from Red Hat today. Origin is a great way to get started with Kubernetes, and what better place to run a container orchestration layer than on top of Fedora Atomic Host?

Kompose Up for OpenShift and Kubernetes

Thu, 01 Dec 2016 00:00:00 +0000

Cross posted with this_ Red Hat Developer Blog post

Introduction

Kompose is a tool to convert from higher level abstractions of application definitions into more detailed Kubernetes artifacts. These artifacts can then be used to bring up the application in a Kubernetes cluster. What higher level application abstraction should kompose use?

One of the most popular application definition formats for developers is the docker-compose.yml format for use with docker-compose that communicates with the docker daemon to bring up the application. Since this format has gained some traction we decided to make it the initial focus of Kompose to support converting this format to Kubernetes. So, where you would choose docker-compose to bring up the application in docker, you can use kompose to bring up the same application in Kubernetes, if that is your preferred platform.

Fedora 25 available in DigitalOcean

Tue, 29 Nov 2016 00:00:00 +0000

Cross posted with this_ fedora magazine post

Last week the Fedora Project released Fedora 25. This week Fedora Project Community members have worked with the DigitalOcean team to make Fedora 25 available on their platform. If you're not familiar with DigitalOcean already, it's a dead simple cloud hosting platform that's great for developers.

Important Notes

The image has some specific differences from others that Fedora ships. You may need to know about these differences before you use the image.

Sharing a Go library to Python (using CFFI)

Tue, 13 Sep 2016 00:00:00 +0000

Disclaimer

I am not a Go expert so I may not be able to answer questions you may have about this process. I am simply trying to reproduce and document what I saw. Additionally, it was actually recommended to not do this for long running processes because you'll have two runtimes that may work against each other eventually (garbage collection, etc). Be wary.

Booting Lenovo T460s after Fedora 24 Updates

Mon, 15 Aug 2016 00:00:00 +0000

Introduction

I recently picked up a new Lenovo T460s work laptop. It is fairly light and has 20G of memory, which is great for running Virtual Machines. One of the first things I did on this new laptop was install Fedora 24 onto the system. After installing from the install media I was up and running and humming along.

Non Deterministic docker Networking and Source Based IP Routing

Wed, 25 May 2016 00:00:00 +0000

Introduction

In the open source docker engine a new networking model was introduced in docker 1.9 which enabled the creation of separate "networks" for containers to be attached to. This, however, can lead to a nasty little problem where a port that is supposed to be exposed on the host isn't accessible from the outside. There are a few bug reports that are related to this issue.

Fedora BTRFS+Snapper - The Fedora 24 Edition

Sat, 23 Apr 2016 00:00:00 +0000

History

In the past I have configured my personal computers to be able to snapshot and rollback the entire system. To do this I am leveraging the BTRFS filesystem, a tool called snapper, and a patched version of Fedora's grub2 package. The patches needed from grub2 come from the SUSE guys and are documented well in this git repo.

Vagrant: Sharing Folders with vagrant-sshfs

Tue, 05 Apr 2016 00:00:00 +0000

cross posted from this_ fedora magazine post

Introduction

We're trying to focus more on developer experience in the Red Hat ecosystem. In the process we've started to incorporate the Vagrant into our standard offerings. As part of that effort, we're seeking a shared folder solution that doesn't include a bunch of if/else logic to figure out exactly which one you should use based on the OS/hypervisor you use under Vagrant.

802.11ac on Linux With NetGear A6100 (RTL8811AU) USB Adapter

Sun, 24 Jan 2016 00:00:00 +0000

UPDATE - 2018-02-01: I was informed by @nmccrina that the repo linked to in this post is no longer maintained. Please use https://github.com/paspro/rtl8812au instead.

NOTE: Most of the content from this post comes from a blog post I found that concentrated on getting the driver set up on Fedora 21. I did mostly the same steps with a few tweaks.

The CentOS CI Infrastructure: A Getting Started Guide

Sat, 23 Jan 2016 00:00:00 +0000

Background

The CentOS community is trying to build an ecosystem that fosters and encourages upstream communities to continuously perform integration testing of their code running on the the CentOS platform. The CentOS community has built out an infrastructure that (currently) contains 256 servers ("bare metal" servers") that are pooled together to run tests that are orchestrated by a frontend Jenkins instance located at ci.centos.org.

Running Nulecules in Openshift via oc new-app

Mon, 18 Jan 2016 00:00:00 +0000

Intro

As part of the Container Tools team at Red Hat I'd like to highlight a feature of Atomic App: support for execution via OpenShift's cli command oc new-app.

The native support for launching Nulecules means that OpenShift users can easily pull from a library of Atomic Apps (Nuleculized applications) that exist in a Docker registry and launch them into OpenShift. Applications that have been packaged up in a Nulecule offer a benefit to the packager and to the deployer of the application. The packager can deliver one Nulecule to all users that supports many different platforms and the deployer gets a simplified delivery mechanism; deploying a Nulecule via Atomic App is easier than trying to manage provider definitions.

Archived-At Email Header From Mailman 3 Lists

Sun, 10 Jan 2016 00:00:00 +0000

By now most Fedora email lists have been migrated to Mailman3. One little (but killer) new feature that I recently discovered was that Mailman3 includes the RFC 5064 Archived-At header in the emails.

This is a feature I have wanted for a really long time; to be able to find an email in your Inbox and copy and paste a link to anyone without having to find the message in the online archive is going to save a lot of time and decrease some latency when chatting on IRC or some other form of real time communication.

Fedora Cloud Vagrant Boxes in Atlas

Sun, 08 Nov 2015 00:00:00 +0000

Cross posted with this_ fedora magazine post

Since the release of Fedora 22, Fedora began creating Vagrant boxes for cloud images in order to make it easier to set up a local environment for development or testing. In the Fedora 22 release cycle we worked out quite a few kinks and we are again releasing libvirt and virtualbox Vagrant boxes for Fedora 23.

Fedora 23: In the Ocean Again

Fri, 06 Nov 2015 00:00:00 +0000

Cross posted with this_ fedora magazine post

This week was the release week for Fedora 23, and the Fedora Project has again worked together with the DigitalOcean team to make Fedora 23 available in their service. If you're not familiar with DigitalOcean already, it is a dead simple cloud hosting platform which is great for developers.

Using Fedora on DigitalOcean

There are a couple of things to note if you are planning on using Fedora on DigitalOcean services and machines.

Getting Ansible Working on Fedora 23

Thu, 05 Nov 2015 00:00:00 +0000

Cross posted with this_ fedora magazine post

Inspired mostly from a post_ by Lars Kellogg-Stedman.

Intro

Ansible is a simple IT automation platform written in python that makes your applications and systems easier to deploy. It has become quite popular over the past few years but you may hit some trouble when trying to run Ansible on Fedora 23.

kubernetes skydns setup for testing on a single node

Wed, 23 Sep 2015 00:00:00 +0000

Intro

Kubernetes is (currently) missing an integrated dns solution for service discovery. In the future it will be integrated into kubernetes (see PR11599) but for now we have to setup skydns manually.

I have seen some tutorials on how to get skydns working, but almost all of them are rather involved. However, if you just want a simple setup on a single node for testing then it is actually rather easy to get skydns set up.

F23 Cloud Base Test Day September 8th!

Wed, 02 Sep 2015 00:00:00 +0000

cross posted from this_ fedora magazine post

Hey everyone! Fedora 23 has been baking in the oven. The Fedora Cloud WG has elected to do a temperature check on September 8th.

For this test day we are going to concentrate on the base image. We will have vagrant boxes (see this page for how to set up your machine), qcow images, raw images, and AWS EC2 images. In a later test day we will focus on the Atomic images and Docker images.

Installing/Starting Systemd Services Using Cloud-Init

Mon, 03 Aug 2015 00:00:00 +0000

Intro

Using cloud-init to bootstrap cloud instances and install custom sofware/services is common practice today. One thing you often want to do is install the software, enable it to start on boot, and then start it so that you don't have to reboot in order to go ahead and start using it.

The Problem

Actually starting a service can be tricky though because when executing cloud-init configuration/scripts you are essentially already within a systemd unit while you try to start another systemd unit.

Fedora BTRFS+Snapper PART 2: Full System Snapshot/Rollback

Sun, 19 Jul 2015 00:00:00 +0000

History

In part 1 of this series I discussed why I desired a computer setup where I can do full system snapshots so I could seamlessly roll back at will. I also gave an overview of how I went about setting up a system so it could take advantage of BTRFS and snapper to do full system snapshotting and recovery. In this final post of the series I will give an overview of how to get snapper installed and configured on the system and walk through using it to do a rollback.

Fedora BTRFS+Snapper PART 1: System Preparation

Tue, 14 Jul 2015 00:00:00 +0000

The Problem

For some time now I have wanted a linux desktop setup where I could run updates automatically and not worry about losing productivity if my system gets hosed from the update. My desired setup to achieve this has been a combination of snapper and BTRFS, but unfortunately the support on Fedora for full rollback isn't quite there.

Encrypting More: /boot Joins The Party

Mon, 06 Jul 2015 00:00:00 +0000

Typically when installing major linux distros they make it easy to select encryption as an option to have encrypted block devices. This is great! The not so great part is the linux kernel and the initial ramdisk aren't typically invited to the party; they are left sitting in a separate and unencrypted /boot partition. Historically it has been necessary to leave /boot unencrypted because bootloaders didn't support decrypting block devices. However, there are some dangers to leaving the bootloader and ramdisks unencrypted (see this post).

Atomic Host Red Hat Summit Lab

Fri, 26 Jun 2015 00:00:00 +0000

Red Hat Summit was a blast this year. I participated in several Hands On Labs to help the community learn about the new tools that are available in the ecosystem. For one of the labs I wrote up a section on Atomic Host, but more specifically on rpm-ostree. I have copied a portion of the lab here as well as added example text to the code blocks.

Fedora 22 Updates-Testing Atomic Tree

Tue, 23 Jun 2015 00:00:00 +0000

It has generally been difficult to test new updates for the rpm-ostree or ostree packages for Atomic Host. This is because in the past you had to build your own tree in order to test them. Now, however, Fedora has starting building a tree based off the updates-testing yum repositories. This means that you can easily test updates by simply running Fedora Atomic Host and rebasing to the fedora-atomic/f22/x86_64/testing/docker-host ref:

Fedora 22 Now Swimming in DigitalOcean

Fri, 05 Jun 2015 00:00:00 +0000

cross posted from this_ fedora magazine post

DigitalOcean is a cloud provider that provides a one-click deployment of a Fedora Cloud instance to an all-SSD server in under a minute. After some quick work by the DigitalOcean and Fedora Cloud teams we are pleased to announce that you can now make it rain Fedora 22 droplets!

One significant change over previous Fedora droplets is that this is the first release to have support for managing your kernel internally. Meaning if you dnf update kernel-core and reboot then you'll actually be running the kernel you updated to. Win!

F22 Cloud/Atomic Test Day May 7th!

Mon, 04 May 2015 00:00:00 +0000

Hey everyone! Fedora 22 is on the cusp of being released and the Fedora Cloud Working Group has elected to organize a test day for May 7th in order to work out some bugs before shipping it off to the rest of the world.

With a new release comes some new features and tools. We are working on Vagrant images as well as a testing tool called Tunir. Joe Brockmeier has a nice writeup about Vagrant and Kushal Das maintains some docs on Tunir.

Crisis Averted.. I'm using Atomic Host

Sat, 31 Jan 2015 00:00:00 +0000

This blog has been running on Docker on Fedora 21 Atomic Host since early January. Occasionally I log in and run rpm-ostree upgrade followed by a subsequent reboot (usually after I inspect a few things). Today I happened to do just that and what did I come up with?? A bunch of 404s. Digging through some of the logs for the systemd unit file I use to start my wordpress container I found this:

quick audit rules for sanity check

Mon, 26 Jan 2015 00:00:00 +0000

Most of the time when I really want to figure out what is going on deep within a piece of software I break out strace and capture all the gory detail. Unfortunately it isn't always that easy to manipulate and run something from the command line but I have found that some simple uses of the audit daemon can give you great insight without having to dig too deep.

Fedora 21 now available on Digital Ocean

Mon, 12 Jan 2015 00:00:00 +0000

cross posted from this_ fedora magazine post

It's raining Droplets! Fedora 21 has landed in Digital Ocean's cloud hosting. Fedora 21 offers a fantastic cloud image for developers, and it's now easy for Digital Ocean users to spin it up and get started! Here are a couple of tips:

Like with other Digital Ocean images, you will log in with your ssh key as root rather than the typical fedora user that you may be familiar with when logging in to a Fedora cloud image.
This is the first time Digital Ocean has SELinux enabled by default (yay for security). If you want or need to you can still easily switch back to permissive mode; Red Hat's Dan Walsh may have a "shame on you" or two for you though.
Fedora 21 should be available in all the newest datacenters in each region, but some legacy datacenters aren't supported. If you have a problem you think is Fedora specific then drop us an email at cloud@lists.fedoraproject.org, ping us in #fedora-cloud on freenode, or visit the Fedora cloud trac to see if it is already being worked on.

Happy Developing!

Dusty

PS If anyone wants a $10 credit for creating a new account you can use my referral link

qemu-img Backing Files: A Poor Man's Snapshot/Rollback

Sun, 11 Jan 2015 00:00:00 +0000

I often like to formulate detailed steps when trying to reproduce a bug or a working setup. VMs are great for this because they can be manipulated easily. To manipulate their disk images I use qemu-img to create new disk images that use other disk images as a backing store. This is what I like to call a "poor man's" way to do snapshots because the snapshotting process is a bit manual, but that is also why I like it; I don't touch the original disk image at all so I have full confidence I haven't compromised it.

Capture Elusive cloud-init Debug Output With journalctl

Mon, 08 Sep 2014 00:00:00 +0000

Recently I have been trying to debug some problems with cloud-init in the alpha versions of cloud images for CentOS 7 and Fedora 21. What I have found is that it’s not so straight forward to figure out how to set up debug logging.

The defaults (defined in /etc/cloud/cloud.cfg.d/05_logging.cfg ) for some reason don’t really capture the debug output in /var/log/cloud-init.log. Luckily, though, on systemd based systems we can get most of that output by using journalctl. There are several services releated to cloud-init and if you want to get the output from all of them you can just use wildcard matching in journalctl (freshly added in ea18a4b ) like so:
\

Docker: Copy Into A Container Volume

Wed, 27 Aug 2014 00:00:00 +0000

I need to copy a few files into my docker container.. Should be easy right? Turns out it’s not so trivial. In Docker 1.0.0 and earlier the docker cp command can be used to copy files from a container to the host, but not the other way around…

Most of the time you can work around this by using an ADD statement in the Dockerfile but I often need to populate some data within data-only volume containers before I start other containers that use the data. To achieve copying data into the volume you can simply use tar and pipe the contents into the volume within a new container like so:\

Creating Your Own Minimal Docker Image in Fedora

Mon, 07 Jul 2014 00:00:00 +0000

Sometimes it can be useful to have a docker image with just the bare essentials. Maybe you want to have a container with just enough to run your app or you are using something like data volume containers and want just enough to browse the filesystem. Either way you can create your own minimalist busybox image on Fedora with a pretty simple script.

The script below was inspired a little from Marek Goldmann’s post about creating a minimal image for wildfly and a little from the busybox website .
\

Manual Linux Installs with Funky Storage Configurations

Thu, 29 May 2014 00:00:00 +0000

Introduction

I often find that my tastes for hard drive configurations on my installed systems is a bit outside of the norm. I like playing with thin LVs, BTRFS snapshots, or whatever new thing there is around the corner. The Anaconda UI has been adding support for these fringe cases but I still find it hard to get Anaconda to do what I want in certain cases.

An example of this happened most recently when I went to reformat and install Fedora 20 on my laptop. Ultimately what I wanted was encrypted root and swap devices and btrfs filesystems on root and boot. One other requirement was that I needed to leave sda4 (a Windows Partition) completely intact. At the end the configuration should look something like:
\

TermRecord: Terminal Screencast in a Self-Contained HTML File

Mon, 19 May 2014 00:00:00 +0000

Introduction

Some time ago I wrote a few posts ( 1, 2 ) on how to use script to record a terminal session and then scriptreplay to play it back. This functionality can be very useful by enabling you the power to show others what happens when you do insert anything here.

I have been happy with this solution for a while until one day Wolfgang Richter commented on my original post and shared a project he has been working on known as TermRecord.

I gave it a spin and have been using it quite a bit. Sharing a terminal recording now becomes much easier as you can simply email the .html file or you can host it yourself and share links. As long the people you are sharing with have a browser then they can watch the playback. Thus, it is not tied to a system with a particular piece of software and clicking a link to view is very easy to do :)

Zero to Wordpress on Docker in 5 Minutes

Sat, 10 May 2014 00:00:00 +0000

Introduction

Docker is an emerging technology that has garnered a lot of momentum in the past year. I have been busy with a move to NYC and a job change (now officially a Red Hatter), so I am just now getting around to getting my feet wet with Docker.

Last night I sat down and decided to bang out some steps for installing wordpress in a docker container. Eventually I plan to move this site into a container so I figured this would be a good first step.

Fedup 19 to 20 with a Thin LVM Configuration

Sun, 04 May 2014 00:00:00 +0000

Introduction

I have been running my home desktop on thin logical volumes for a while now. I have enjoyed the flexibility of this setup and I like taking a snapshot before making any big changes to my setup. Recently I decided to update to Fedora 20 from Fedora 19 and I hit some trouble along the way because the Fedora 20 initramfs (images/pxeboot/upgrade.img) that is used by fedup for the upgrade does not have support for thin logical volumes. After running fedup and rebooting you end up with a message to the screen that looks something like this:\

Nested Virt and Fedora 20 Virt Test Day

Mon, 21 Oct 2013 00:00:00 +0000

Introduction

I decided this year to take part in the Fedora Virtualization Test Day on October 8th. In order to take part I needed a system with Fedora 20 installed so that I could then create VMs on top. Since I like my current setup and I didn’t have a hard drive laying around that I wanted to wipe I decided to give nested virtualization a shot.

Most of the documentation I have seen for nested virtualization has come from Kashyap Chamarthy. Relevant posts are here, here, and here. He has done a great job with these tutorials and this post is nothing more than my notes for what I found to work for me.

BTRFS: How big are my snapshots?

Sun, 22 Sep 2013 00:00:00 +0000

Introduction

I have been using BTRFS snapshots for a while now on my laptop to incrementally save the state of my machine before I perform system updates or run some harebrained test. I quickly ran into a problem though, as on a smaller filesystem I was running out of space. I then wanted to be able to look at each snapshot and easily determine how much space I could recover if I deleted each snapshot. Surprisingly this information was not readily available. Of course you could determine the total size of each snapshot by using du, but that only tells you how big the entire snapshot is and not how much of the snapshot is exclusive to this snapshot only..

Enter filesystem quota and qgroups in git commit 89fe5b5f666c247aa3173745fb87c710f3a71a4a . With quota and qgroups (see an overview here ) we can now see how big each of those snapshots are, including exclusive usage.

Excellent LVM Tutorial for Beginners or Experts

Sun, 15 Sep 2013 00:00:00 +0000

I ran across a great PDF from this year’s Red Hat Summit in Boston. Hosted by Christoph Doerbech and Jonathan Brassow the lab covers the following topics:\

What is LVM? What are filesystems? etc..
Creating PVs, VGs, LVs.
LVM Striping and Mirroring.
LVM Raid.
LVM Snapshots (and reverting).
LVM Sparse Volumes (a snapshot of /dev/zero).
LVM Thin LVs and new snapshots.

Check out the PDF here . If that link ceases to work at some point I have it hosted here as well.

Hope everyone can use this as a great learning tool!

Dusty

Convert an Existing System to Use Thin LVs

Sat, 07 Sep 2013 00:00:00 +0000

Introduction

Want to take advantage of the efficiency and improved snapshotting of thin LVs on an existing system? It will take a little work but it is possible. The following steps will show how to convert a CentOS 6.4 basic installation to use thin logical volumes for the root device (containing the root filesystem).

Preparation

To kick things off there are few preparation steps we need that seem a bit unreleated but will prove useful. First I enabled LVM to issue discards to underlying block devices (if you are interested in why this is needed you can check out my post here. )
\

RPM File Colors

Sun, 25 Aug 2013 00:00:00 +0000

What are RPM file colors?

When building a package rpm will tag each file within the package with a file color. Usually the file color will fit into one of four categories as described by Jeff Johnson here. These categories are:\

0 is unknown or other
1 is Elf32
2 is Elf64
3 is (contains) both

So why does rpm do this? The short answer is “for multilib support”. Basically so we can install both the 32bit and 64bit version of a package on the system and have some hopes of everything still working correctly.

An example of this would be a system that needed both the 32bit and 64bit version of glibc (chances are you have them both installed because some package has been slow to move to 64bit and only provides software compiled for 32bit). The problem with having both rpms installed is that both rpms provide some of the same files (i.e /sbin/ldconfig). Which one should rpm choose? This is where file colors come in.

When installing a file from a new package rpm will check to see if the file is already provided by another rpm and will then use file color to determine if the file should be replaced or left alone. The current behavior of rpm is to prefer 64bit over 32bit files. That means when both i686 and x86_64 glibc are installed, ldconfig should be 64bit. This can easily be checked:
\

Thin LVM Snapshots: Why Size Is Less Important

Wed, 17 Jul 2013 00:00:00 +0000

Traditionally with LVM snapshots you need to be especially careful when choosing how big to make your snapshots; if it is too small it will fill up and become invalid. If taking many snapshots with limited space then it becomes quite difficult to decide which snapshots need more space than others.

One approach has been to leave some extra space in the VG and let dmeventd periodically poll and lvextend the snapshot if necessary (I covered this in a previous post ). However, a reader of mine has pointed out that this polling mechanism does not work very well for small snapshots.

Fortunately, with the addition of thin logical volume support within LVM (I believe initially in RHEL/CentOS 6.4 and/or Fedora 17), size is much less important to consider when taking a snapshot. If you create a thin LV and then “snapshot” the thin LV, what you actually end up with are two thin LVs. They both use extents from the same pool and the size will grow dynamically as needed.

As always, examples help. In my system I have a 20G sdb. I’ll create a VG, vgthin, that uses sdb and then a 10G thin pool, lvpool, within vgthin.
\

Find Guest IP address using QEMU Guest Agent

Sun, 14 Jul 2013 00:00:00 +0000

Ever needed to find the IP address of a particular guest? Of course, the answer is “yes”. For the most part I have either resorted to going in through the console of the VM to find this information or used some nifty little script like the one described here by Richard Jones. However, if you have qemu Guest Agent set up ( I covered this briefly in a previous post ), then you can just query this information using the guest-network-get-interfaces qemu-ga command:
\

Enabling QEMU Guest Agent anddddd FSTRIM (AGAIN)

Wed, 26 Jun 2013 00:00:00 +0000

In an earlier post I walked through reclaiming disk space from guests using FSTRIM and in a follow up I showed how to do the same thing with thin Logical Volumes as the sparse backing storage for the disk images. In both of the previous posts I logged in to the guest first and then executed the fstrim command in order to release the free blocks back to the underlying block devices.

Thankfully, due to some recent work, this operation has now been exposed externally via qemu Guest Agent and can be executed remotely via libvirt. To enable qemu Guest Agent, first I added a virtio-serial device that the host and guest will use for communication. I did this by adding the following to the guest’s xml:
\

Guest Discard/FSTRIM On Thin LVs

Fri, 21 Jun 2013 00:00:00 +0000

In my last post I showed how to recover space from disk images backed by sparse files. As a small addition I’d like to also show how to do the same with a guest disk image that is backed by a thinly provisioned Logical Volume.

First things first, I modified the /etc/lvm/lvm.conf file to have the issue_discards = 1 option set. I’m not 100% sure this is needed but I did it at the time so I wanted to include it here.

Next I created a new VG (vgthin) out of a spare partition and then created an thin LV pool (lvthinpool) inside the VG. Finally I created a thin LV within the pool (lvthin). This is all shown below:
\

Recover Space From VM Disk Images By Using Discard/FSTRIM

Tue, 11 Jun 2013 00:00:00 +0000

Sparse guest disk image files are a dream. I can have many guests on a small amount of storage because they are only using what they need. Of course, if each guest were to suddenly use all of the space in their filesystems then the host filesystem containing the guest disk images would fill up as well. However, since filesystems grow over time rather than overnight, with proper monitoring you can foresee this event and add more storage as needed.

Sparse guest disk images aren’t all bells and whistles though. Over time files are created/deleted within the filesystems on the disk images and the images themselves are no longer as compact as they were in the past. There is good news though; we can recover the space from all of those deleted files!

Easy getopt for a BASH script

Fri, 17 May 2013 00:00:00 +0000

getopt is extremely useful for quickly being able to add options and arguments to your program without having to worry much about the parsing yourself. There are getopt libraries for many languages but what about BASH? It turns out there are actually two versions of getopt that you can use in your BASH scripts; a command line utility getopt provided by the util-linux package, and a bash builtin getopts. I have provided a brief overview of each in the following sections.

Booting Anaconda from Software RAID1 Device

Sun, 05 May 2013 00:00:00 +0000

Sometimes you just want to boot Anaconda from a software raid device that houses both the stage1 (initrd.img) and stage2 (install.img) images. There are various reasons to do this some of which include:

Booting Anaconda into rescue mode from hard drive (RAID 1)
Installing directly from hard drive (RAID 1)
Running PreUpgrade (Now Deprecated)

Running Anaconda from a RAID 1 device is unsupported at least up until the rhel 6.4 version of Anaconda and is documented in BZ #500004 (note that this is a “feature”, not a bug) . It may be supported now with the new Anaconda redesign , but I havn’t put in the time to find out yet.

That being said, a good workaround mentioned in the comments of the bug report is to simply attach an install cd that contains the same version of Anaconda you are trying to boot and it will automatically find it and continue on. However, if you often need to boot Anaconda like this it can be tedious and there is another way that may be more desireable to some.

Since /boot/ is set up on a software RAID 1 each member device contains an exact copy of what is on the RAID device. This means that you can directly mount any of the member devices as long as you specify the filesystem type. This is exactly where Anaconda has a problem.

The kernel and stage1 are loaded by grub, which sucessfully ignores the fact that the device is a part of a raid and just treats it like an ext filesystem. Anaconda on the other hand attempts to mount the specified device to find the stage2 image. In doing this Anaconda calls /bin/mount and specifies an fs type of auto ( -t auto ). Since the device has an MD superblock, mount fails to detect it is ext4 and does not mount the device.

What is the solution for this?? Well, we need to get rid of the superblock :)

As a brief example I will show how to set up Anaconda to boot into rescue mode from a software RAID 1. First we need to copy the kernel, stage1, and stage2 images from an install cd and into the /boot/ filesystem.
\

OS Upgrade and Rollback Using BTRFS

Sun, 06 Jan 2013 00:00:00 +0000

I recently decided to try out the snapshotting capabilities of the relatively new BTRFS filesystem. I have been using the snapshot and rollback capability of LVM (using lvconvert --merge) for a while now so I figured I would check out BTRFS to see how it stacks up.

To get up to speed on how to use BTRFS I found the BTRFS Fun web page a good reference. I converted an existing Fedora 17 virtual machine to use BTRFS for the filesystems (I may cover how I did this in a later post). The disk inside the virtual machine contains three partitions as is shown below:
\

Running Benchmarks With the Phoronix Test Suite!

Sun, 30 Dec 2012 00:00:00 +0000

Benchmarking software can be invaluable when testing new hardware/software configurations. The Phoronix Test Suite is a collection of open source software benchmarks that are fairly easy to use and the results are presented in a such a way that is easy to understand; even if you don’t understand the tests that were run. Today I’ll give a brief run down of how to install the test suite and run the benchmarks.

Phoronix can be used on almost all operating systems. The only requirement of the Phoronix Test Suite is PHP. For this demonstration I am using a Fedora 17 virtual machine, however your experience should be fairly similar on whatever Linux distribution you may be using.

The first thing I needed to do was install a few PHP packages. A quick call to yum will take care of this for us:\

Trace Function Calls Using GDB Revisited!

Mon, 17 Dec 2012 00:00:00 +0000

In an earlier post I discussed how to trace calls using GDB so that function calls and their arguments can easily be viewed. What I neglected to mention was rbreak , a feature of GDB to be able to set breakpoints using a regular expression.

Using rbreak you can get the same functionality but with much less effort. For example, to get the same behavior as before (setting a breakpoint on each function call and printing a trace of the bottom most level) all you need to provide to GDB are the following commands:
\

Mount Complex Disk Images Using libguestfs

Sun, 16 Dec 2012 00:00:00 +0000

In my previous post I went over two ways to mount a partition within a disk image file. There is actually another easy way to do this by utilizing some of the tools that the virt community has provided us in recent years.

libguestfs is a fairly comprehensive library for manipulating guest disk images and filesystems. It turns out that the tools work pretty well even for disk images that aren’t specific to any virtualized guest; after all a disk image is a disk image.

The guestmount utility is provided as part of the libguestfs-tools-c package in Fedora 17 and allows us the ability to (in)directly mount the second partition of our disk image. This is shown below:
\

Mounting a Partition Within a Disk Image

Sat, 15 Dec 2012 00:00:00 +0000

Last time I walked through creating a sparse disk image using dd and cp --sparse=always. OK, we have a disk image. Now what?

Normally it would suffice to just set up a loop device and then mount, but this disk image doesn’t just contain a filesystem. It has 4 partitions each with their own filesystem. This means in order to mount one of the filesystems we have to take a few extra steps.

There is an easy and a hard way to do this. I’ll start with the hard way..\

Create A Disk Image Without Enough Free Space

Thu, 15 Nov 2012 00:00:00 +0000

Recently I purchased a new laptop. Typically my first move is to ditch Windows or Mac (yes I’ve done both) and install Linux. This time, just in case I ever want to completely restore the system (recovery partition and all), I decided to make a disk image of the entire hard drive.

Being that the capacity of the hard drive is close to 500 GB it actually turned out that I didn’t have enough free space to store the entire image. However, since it was a brand new computer I knew there was plenty of zeroes on the disk and thus plenty of opportunity to reduce the actual size of the disk image by converting those zeroes into holes in a sparse file .

Now usually when creating a disk image I would just use dd, but dd doesn’t have any options that are sparse file aware. Luckily I was able to find a page that addressed my particular predicament.

The secret sauce here is the --sparse=always option of cp. By piping the output of dd to cp, I could then copy my disk image! The next step was to fire up a live cd, connect an external hard drive and run the following command to create the disk image:\

Trace Function Calls Using GDB

Sun, 14 Oct 2012 00:00:00 +0000

Sometimes it is easier to debug when you are able to view a call trace of all function calls in a particular program. This is especially true when working with code that isn’t yours or when debugging issues such as infinite loops in your own code. The way I typically do this is by creating a GDB commands file that defines breakpoints for each function I would like to see in the trace.

For each function breakpoint, I instruct GDB to print a short backtrace and then continue execution. I can then stop the program run at any time using CTRL-C and observe where the program is, what functions are being called, and what arguments they are being called with.

To illustrate this, consider the following short program that accepts two arguments. It takes these two arguments, squares them, and then finds the lowest common multiple of their squares. This means if you pass in 4 and 10, it will find the LCM of 16 and 100.

NOTE: I have defined functions for basic operations such as squaring a number and adding two numbers just for illustrative purposes.
\

Share a Folder Between KVM Host and Guest

Tue, 11 Sep 2012 00:00:00 +0000

I often find myself in situations where I need to share information or files between a KVM host and KVM guest. With libvirt version 0.8.5 and newer there is support for mounting a shared folder between a host and guest. I decided to try this out on my Fedora 17 host, with a Fedora 17 guest.

Using the libvirt <filesystem> xml tag I created the following xml that defines a filesystem device.
\

Exchanging SSH keys using ssh-copy-id

Sat, 18 Aug 2012 00:00:00 +0000

It is common practice among Linux users to exchange ssh keys between machines so that you can ssh between them without having to authenticate. The manual process for doing this involves taking the public key of the local host (~/.ssh/id_rsa.pub or ~/.ssh/id_dsa.pub) and appending it to the ~/.ssh/authorized_keys file of the remote host you wish to log in without a password.

This process is simple, but requires a few different steps. Luckily, openssh includes a nifty little shell script that will take care of all of the manual steps for you. This script is called ssh-copy-id and should be available on your Linux distro as long as you are using the openssh client.

In order to use it all you need to do is provide a username and a remote host to log in to. It will then copy your key to the authorized_keys file of the remote host and from then on you should be able to log in without authenticating. This is illustrated below:
\

Easily Resize LVs and Underlying Filesystems

Sun, 13 May 2012 00:00:00 +0000

Part of the reason I use Logical Volumes for my block devices rather than standard partitions is because LVs are much more flexible when it comes to sizing/resizing.

For example, in a particular setup you might have a 1 TB hard drive that you want to be broken up into two block devices. You could either choose two 500 GB partitions, or two 500 GB LVs. If you use partitions and later find out that you really needed 300 GB for one and 700 GB for the other then resizing might get a little complicated. On the other hand, with LVs resizing is simple!

LVM has the ability to resize the LV and the underlying filesystem at the same time (it uses fsadm under the covers to resize the filesystem which on my system supports resizing ext2/ext3/ext4/ReiserFS/XFS). In order to pull this off simply use lvresize along with the --resizefs option. An example of this command is shown below:
\

Send Magic SysRq to a KVM guest using virsh

Sat, 21 Apr 2012 00:00:00 +0000

When a linux computer is “hung” or “frozen” you can use a Magic SysRq key sequence to send various low level requests to the kernel in order to try to recover from or investigate the problem. This is extremely useful when troubleshooting server lockups, but until recently libvirt did not expose this functionality for KVM guests.

In v0.9.3 (and newer) of libvirt you can send a Magic SysRq sequence to a guest by utilizing the send-key subcommand provided by virsh. In other words, sending the ‘h’ Magic SysRq command is as simple as:
\

Terminal Screencast Revisit: Log output from a multiplexed terminal

Sun, 15 Apr 2012 00:00:00 +0000

Traditionally with script you have really only been able to log the output of a simple “single session” terminal window. However, if you capture the timing data from script you can actually log much more than that.

In a previous post I talked about how to save the timing data using script and then play it back using scriptreplay. If you use this technique and also use a a terminal multiplexer such as screen or tmux you can capture much more information in your screencast.

To see an example of this you can watch the video below where I use scriptreplay to play back an earlier recorded screencast.
\

Hard Drive Monitoring and E-mail Alerts Using smartd

Sat, 10 Mar 2012 00:00:00 +0000

A while back I set up mdadm to monitor my RAID array and send email alerts to notify me of failures. At the same time I also set up smartd (see S.M.A.R.T. ) to monitor the hard drives themselves and to send me email alerts.

To do this you edit the /etc/smartd.conf file. After I was done my /etc/smartd.conf file looked like the following:

#
# HERE IS A LIST OF DIRECTIVES FOR THIS CONFIGURATION FILE.
# PLEASE SEE THE smartd.conf MAN PAGE FOR DETAILS
#
#   -d TYPE Set the device type: ata, scsi, marvell, removable, 3ware,N, hpt,L/M/N
#   -T TYPE set the tolerance to one of: normal, permissive
#   -o VAL  Enable/disable automatic offline tests (on/off)
#   -S VAL  Enable/disable attribute autosave (on/off)
#   -n MODE No check. MODE is one of: never, sleep, standby, idle
#   -H      Monitor SMART Health Status, report if failed
#   -l TYPE Monitor SMART log.  Type is one of: error, selftest
#   -f      Monitor for failure of any 'Usage' Attributes
#   -m ADD  Send warning email to ADD for -H, -l error, -l selftest, and -f
#   -M TYPE Modify email warning behavior (see man page)
#   -s REGE Start self-test when type/date matches regular expression (see man page)
#   -p      Report changes in 'Prefailure' Normalized Attributes
#   -u      Report changes in 'Usage' Normalized Attributes
#   -t      Equivalent to -p and -u Directives
#   -r ID   Also report Raw values of Attribute ID with -p, -u or -t
#   -R ID   Track changes in Attribute ID Raw value with -p, -u or -t
#   -i ID   Ignore Attribute ID for -f Directive
#   -I ID   Ignore Attribute ID for -p, -u or -t Directive
#   -C ID   Report if Current Pending Sector count non-zero
#   -U ID   Report if Offline Uncorrectable count non-zero
#   -W D,I,C Monitor Temperature D)ifference, I)nformal limit, C)ritical limit
#   -v N,ST Modifies labeling of Attribute N (see man page)
#   -a      Default: equivalent to -H -f -t -l error -l selftest -C 197 -U 198
#   -F TYPE Use firmware bug workaround. Type is one of: none, samsung
#   -P TYPE Drive-specific presets: use, ignore, show, showall
#    #      Comment: text after a hash sign is ignored
#    \      Line continuation character
# Attribute ID is a decimal integer 1 <= ID <= 255
# except for -C and -U, where ID = 0 turns them off.
# All but -d, -m and -M Directives are only implemented for ATA devices
#
# If the test string DEVICESCAN is the first uncommented text
# then smartd will scan for devices /dev/hd[a-l] and /dev/sd[a-z]
DEVICESCAN -o on -H -l error -l selftest -t -m dustymabe@gmail.com -M test

It is pretty much all comments except for the last line. You can see from the comments what each option on the last line means. To summarize I am telling smartd to:

“Monitor the health status as well as the error and selftest logs of all /dev/hd[a-l] and /dev/sd[a-z] devices that are discovered to have SMART capabilities. Report any errors/failures as well as startup test messages to dustymabe@gmail.com.”

Now just make sure the smartd service is configured to run by default and your disks should be monitored! You can check this by looking to see if you get an email when smartd starts (make sure to check your spam filter).

Dusty Mabe

Automatically Extend LVM Snapshots

Sun, 04 Mar 2012 00:00:00 +0000

Snapshot logical volumes are a great way to save the state of an LV (a special block device) at a particular point in time. Essentially this provides the ability to snapshot block devices and then revert them back at a later date. In other words you can rest easy when that big upgrade comes along :)

This all seems fine and dandy until your snapshot runs out of space! Yep, the size of the snapshot does matter. Snapshot LVs are Copy-On-Write (COW) devices. Old blocks from the origin LV get “Copied” to the snapshot LV only when new blocks are “Written” to in the origin LV. Additionally, only the blocks that get written to in the origin LV get copied over to the snapshot LV.

Thus, you can make a snapshot LV much smaller than the origin LV and as long as the snapshot never fills up then you are fine. If it does fill up, then the snapshot is invalid and you can no longer use it.

The problem with this is the fact that it becomes quite tricky to determine how much space you actually need in your snapshot. If you notice that your snapshot is becoming full then you can use lvextend to increase the size of the snapshot, but this is not very desirable as it’s not automated and requires user intervention.

The good news is that recently there was an addition to lvm that allows for autoextension of snapshot LVs! The bugzilla report #427298 tracked the request and it has now been released in lvm2-2.02.84-1. The lvm-devel email from when the patch came through contains some good details on how to use the new functionality.

To summarize, you edit /etc/lvm/lvm.conf and set the snapshot_autoextend_threshold to something other than 100 (100 is the default value and also disables automatic extension). In addition, you also edit the snapshot_autoextend_percent. This value will be the amount you want to extend the snapshot LV.

To test this out I edited my /etc/lvm/lvm.conf file to have the following values:
\

Monitor RAID Arrays and Get E-mail Alerts Using mdadm

Sun, 29 Jan 2012 00:00:00 +0000

In my Desktop computer I use a software RAID1 to protect against a data loss due to a hard drive failure. I have two hard drives, each with four identically sized partitions. Partition 1 on disk A is mirrored with partition 1 on disk B. Together they create the “multiple-device” device node md1 which can then be treated like any block device. Partitions 2, 3, 4 on the disks make up md2, md3, and md4 respectively.

You can use mdadm to configure a software raid in Linux. To see the status of the raid you can view the contents of the /proc/mdstat file. For my software raid the contents of the file should look like:
\

Recover Space By Finding Deleted Files That Are Still Held Open.

Sun, 22 Jan 2012 00:00:00 +0000

The other day I was trying to clean out some space on an almost full filesystem that I use to hold some video files. The output from df looked like:
\

media:~ # df -kh /docs/videos/
Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/vgvolume-videos 5.0G  4.2G  526M  90% /docs/videos

I then found the largest file I wanted to delete (a 700M avi video I had recently watched), and removed it. df should now report that I freed up some space right? NOPE!
\

Create a screencast of a terminal session using scriptreplay.

Wed, 11 Jan 2012 00:00:00 +0000

I recently ran into an issue where I needed to demo a project without actually being present for the demo. I thought about recording (into some video format) a screencast of my terminal window and then having my audience play it at the time of the demo. This would have worked just fine, but, as I was browsing the internet searching for exactly how to record a screencast of this nature, I ran across a blog post talking about how to play back terminal sessions using the output of the script program. This piqued my interest for several reasons:

Hi Planet! - SSH: Disable checking host key against known_hosts file.

Mon, 09 Jan 2012 00:00:00 +0000

Hi Everyone! Since this is my first post it is going to be short and sweet. :)

I work on a daily basis with Linux Servers that must be installed, configured, re-installed, configured etc… Over and over, develop and test. Our primary means of communication with these servers is through ssh. Every time a server is re-installed it generates a new ssh key and thus you will always get a “Man in the Middle Attack” warning from SSH like:
\

Speaking Experience

Mon, 01 Jan 0001 00:00:00 +0000

2023/10 — OpenShift Commons — Raleigh, NC
- Red Hat CoreOS Layering Updates
  - Type: Talk
  - Links: Agenda, Slides, Recording: YouTube, dustymabe.com
2023/10 — All Things Open — Raleigh, NC
- What is Fedora CoreOS?
  - Type: Talk
  - Links: Agenda, Slides
2023/06 — Fedora 38 Release Party — Online, Worldwide
- Fedora CoreOS News & How Columbia University Uses Fedora CoreOS
  - Type: Talk
  - Co-Presenters:
    - Marc Pusey — Chief of Software Engineering & Architecture, Columbia University
  - Links: Agenda, Slides, Recording: YouTube dustymabe.com
2022/08 — DevConf.US — Boston, MA
- Fedora CoreOS: What’s Now, What’s Next
  - Type: Talk
  - Co-Presenters:
    - Mike Nguyen — Senior Software Quality Engineer, Red Hat
  - Links: Agenda, Slides, Recording: YouTube
- Hands On with Fedora CoreOS
  - Type: Workshop/Lab
  - Co-Presenters:
    - Sohan Kunkerkar — Senior Software Engineer, Red Hat
  - Links: Agenda, Slides
2022/08 — Flock/Nest — Online, Worldwide
- Fedora on the World’s Computer - Onboarding Fedora to Microsoft Azure
  - Type: Talk
  - Co-Presenters:
    - Jack Aboutboul — LSG Program Manager, Microsoft
  - Links: Slides, Recording: YouTube dustymabe.com
2022/05 — Fedora 36 Release Party — Online, Worldwide
- Fedora CoreOS
  - Type: Talk
  - Links: Agenda, Slides, Recording: YouTube dustymabe.com
2021/08 — Flock/Nest — Online, Worldwide
- What’s New and What’s Next in Fedora CoreOS
  - Type: Talk
  - Co-Presenters:
    - Timothée Ravier — Senior Software Engineer, Red Hat
  - Links: Agenda, Slides, Recording: YouTube dustymabe.com
2020/11 — KubeCon North America — Online, Worldwide
- Linux in the Kubernetes Era: Does The OS Still Matter?
  - Type: Panel
  - Co-Presenters:
    - Tasha Drew — Director of Product Incubation, VMware
    - Christian “Kiko” Reis — VP Public Cloud, Canonical
    - Darren Shepherd — Chief Technology Officer, Rancher Labs
    - Vincent Batts — Chief Technology Officer, Kinvolk
  - Links: Agenda, Recording: YouTube, dustymabe.com
2020/08 — OKD Working Group — Online, Worldwide
- OKD on Fedora CoreOS on DigitalOcean
  - Type: Talk
  - Co-Presenters:
    - Neal Gompa — Senior DevOps Engineer, Datto
  - Links: Git, Blog, Recording: YouTube, dustymabe.com
2020/08 — Flock/Nest — Online, Worldwide
- Fedora CoreOS: What’s Now, What’s Next
  - Type: Talk
  - Links: Agenda, Slides, Recording: YouTube dustymabe.com
- Getting Started with Fedora CoreOS: A Hands-On Lab
  - Type: Workshop/Lab
  - Co-Presenters:
    - Timothée Ravier — Senior Software Engineer, Red Hat
    - Nasir Hussain — Student
  - Links: Agenda, Slides, Recording: YouTube
2020/05 — Fedora Podcast — Podcast
- Episode 07: Fedora Atomic, Fedora CoreOS and Fedora Silverblue
  - Type: Podcast
  - Links: Recording: fedorapeople, fireside.fm, dustymabe.com
2020/01 — DevConf.CZ — Brno, Czech Republic
- Fedora CoreOS Hands-On
  - Type: Workshop/Lab
  - Co-Presenters:
    - Vadim Rutkovsky — Senior Software Engineer, Red Hat
    - Ben Howard — Principal Software Engineer, Red Hat
  - Links: Agenda
2018/08 — Flock — Dresden, Germany
- State of Fedora Keynote/Objectives & Editions
  - Type: Talk
  - Co-Presenters:
    - Matthew Miller — Fedora Project Leader, Red Hat
  - Links: Agenda, Recording: YouTube
- Building Fedora CoreOS
  - Type: Talk
  - Co-Presenters:
    - Benjamin Gilbert — Senior Software Engineer, Red Hat
  - Links: Agenda, Slides, Recording: YouTube dustymabe.com
2018/06 — LINUX Unplugged — Podcast
- Episode 255: Fedora to the Core
  - Type: Podcast
  - Links: Agenda, Recording: YouTube dustymabe.com
- Catching Up With Atomic: Retrospective & BOF
  - Type: Birds of a Feather
  - Co-Presenters:
    - Josh Berkus — Kubernetes Community Manager, Red Hat
  - Links: Agenda, Slides
2018/05 — Red Hat Summit — San Francisco, CA
- Containerizing Applications: Existing and New
  - Type: Workshop/Lab
  - Co-Presenters:
    - Scott Collier — Senior Principal Systems Engineer, Red Hat
    - Langdon White — Platform Architect, Developer Experience, Red Hat
    - Tommy Hughes — Principal Software Engineer, Red Hat
  - Links: Git
- Atomic BOF
  - Type: Birds of a Feather
  - Co-Presenters:
    - Josh Berkus — Kubernetes Community Manager, Red Hat
2018/01 — DevConf.CZ — Brno, Czech Republic
- Atomic Host 101 Lab
  - Type: Workshop/Lab
  - Links: Agenda
- Catching Up With Atomic: Retrospective & BOF
  - Type: Birds of a Feather
  - Co-Presenters:
    - Josh Berkus — Kubernetes Community Manager, Red Hat
  - Links: Agenda, Slides
2017/11 — TriLUG Lightning Talk — Raleigh, NC
- Minishift
  - Type: Lightning Talk
  - Links: Slides
2017/08 — Flock — Cape Cod, MA
- Atomic Host 101
  - Type: Talk
  - Links: Agenda Recording (no audio): YouTube
- Fedora Atomic Docs Workshop
  - Type: Working Session
  - Links: Agenda
2017/05 — Red Hat Summit — Boston, MA
- Containerizing Applications: Existing and New
  - Type: Workshop/Lab
  - Co-Presenters:
    - Scott Collier — Senior Principal Systems Engineer, Red Hat
    - Langdon White — Platform Architect, Developer Experience, Red Hat
  - Awards: Top Presenter
  - Links: Git
2017/04 — Navops NYC Tech Talk — New York, NY
- Project Atomic and Atomic Host
  - Type: Talk
  - Links: Slides
2016/08 — Flock — Kraków, Poland
- Don’t Destroy Your Machine with Development
  - Type: Talk
  - Links: Agenda, Slides, Recording (no audio): YouTube
- Bringing Developers into the Flock
  - Type: Talk
  - Co-Presenters:
    - Brian Exelbierd — Community Business Owner, Red Hat
  - Links: Agenda, Recording: YouTube dustymabe.com
2016/06 — Red Hat Summit — San Francisco, CA
- Containerizing Applications: Existing and New
  - Type: Workshop/Lab
  - Co-Presenters:
    - Scott Collier — Senior Principal Systems Engineer, Red Hat
    - Langdon White — Platform Architect, Developer Experience, Red Hat
  - Awards: Top Presenter
  - Links: Git
2016/01 — CentOS Dojo — Brussels, Belgium
- The CentOS CI Getting Started Guide
  - Type: Talk
  - Links: Agenda, Slides
2016/01 — FOSDEM — Brussels, Belgium
- The CentOS CI: A Getting Started Guide
  - Type: Talk
  - Links: Description, Slides, Recording: video.fosdem.org dustymabe.com
2015/08 — LinuxCon/ContainerCon — Seattle, WA
- Building Containerized Applications
  - Type: Workshop/Lab
  - Co-Presenters:
    - Daniel Walsh — Senior Distinguished Engineer, Red Hat
    - Bob Kozdemba — Principal Solution Architect, Red Hat
  - Links: Sched
2015/06 — Red Hat Summit — Boston, MA
- Containerizing Applications: Existing and New
  - Type: Workshop/Lab
  - Co-Presenters:
    - Scott Collier — Senior Principal Systems Engineer, Red Hat
    - Aaron Weitekamp — Senior Software Engineer, Red Hat
    - Langdon White — Platform Architect, Developer Experience, Red Hat
  - Awards: Top Presenter
  - Links: Git
- Hands-on Red Hat Enterprise Linux Atomic Host
  - Type: Workshop/Lab
  - Co-Presenters:
    - Bob Kozdemba — Principal Solution Architect, Red Hat
    - Rob Wilmoth — Senior Solutions Architect, Red Hat
  - Links: Blog