Installing/Starting Systemd Services Using Cloud-Init

Intro

Using cloudiinit to bootstrap cloud instances and install custom sofware/services is common practice today. One thing you often want to do is install the software, enable it to start on boot, and then start it so that you don't have to reboot in order to go ahead and start using it.

The Problem

Actually starting a service can be tricky though because when executing cloud-init configuration/scripts you are essentially already within a systemd unit while you try to start another systemd unit.

To illustrate this I decided to start a Fedora 22 cloud instance and install/start docker as part of bringup. The instance I started had the following user-data:

#cloud-config
packages:
  - docker
runcmd:
  - [ systemctl, daemon-reload ]
  - [ systemctl, enable, docker.service ]
  - [ systemctl, start, docker.service ]

After the system came up and some time had passed (takes a minute for the package to get installed) here is what we are left with:

[root@f22 ~]# pstree -asp 925
systemd,1 --switched-root --system --deserialize 21
  `-cloud-init,895 /usr/bin/cloud-init modules --mode=final
      `-runcmd,898 /var/lib/cloud/instance/scripts/runcmd
          `-systemctl,925 start docker.service
[root@f22 ~]# systemctl status | head -n 5
‚óŹ f22
    State: starting
     Jobs: 5 queued
   Failed: 0 units
    Since: Tue 2015-08-04 00:49:13 UTC; 30min ago

Basically the systemctl start docker.service command has been started but is blocking until it finishes. It doesn't ever finish though. As can be seen from the output above it's been 30 minutes and the system is still starting with 5 jobs queued.

I suspect this is because the start command queues the start of the docker service which then waits to be scheduled. It doesn't ever get scheduled, though, because the cloud-final.service unit needs to complete first.

The Solution

Is there a way to get the desired behavior? There is an option to systemctl that will cause it to not block during an operation, but rather just queue the action and exit. This is the --no-block option. From the systemctl man page:

--no-block
    Do not synchronously wait for the requested operation
    to finish. If this is not specified, the job will be
    verified, enqueued and systemctl will wait until it is
    completed. By passing this argument, it is only
    verified and enqueued.

To test this out I just added --no-block to the user-data file that was used previously:

#cloud-config
packages:
  - docker
runcmd:
  - [ systemctl, daemon-reload ]
  - [ systemctl, enable, docker.service ]
  - [ systemctl, start, --no-block, docker.service ]

And.. After booting the instance we get a running service:

[root@f22 ~]# systemctl is-active docker
active

Cheers!

Dusty

8 Responses to “Installing/Starting Systemd Services Using Cloud-Init”


  • Thanks a bunch for the writeup, I just ran into this exact thing and you saved me a ton of wasted time. Much appreciated!

  • Hello, `–no-block` option is interesting but does not resolve the issue. If I want to install docker *and* interact with it all from cloud-init, then that’s not possible as the docker service will start only after cloud-init finish.
    Any workarounds? Very stupid cloud-init setup IMO.

  • Found a workaround – using the –ignore-dependencies. Put this in your user data:

    runcmd:

    – [ systemctl, enable, docker.service ]
    – [ systemctl, start, docker-storage-setup.service, –ignore-dependencies ]
    – [ systemctl, start, docker.service, –ignore-dependencies ]

    Here you can interact with the running docker service.

  • @smoser from #cloud-init also suggested as a possible workaround to create a systemd service using

    bootcmd: []

    And that service would run the things that need interaction with docker.

  • You may find my `dockerfy` utility useful starting services, pre-running initialization commands before the primary command starts. See https://github.com/markriggins/dockerfy

    For example:

    RUN wget https://github.com/markriggins/dockerfy/releases/download/0.2.4/dockerfy-linux-amd64-0.2.4.tar.gz; \
    tar -C /usr/local/bin -xvzf dockerfy-linux-amd64-*tar.gz; \
    rm dockerfy-linux-amd64-*tar.gz;

    ENTRYPOINT dockerfy
    COMMAND –start bash -c “while false; do echo ‘Ima Service’; sleep 1; done” — \
    –reap — \
    nginx

    Would run a bash script as a service, echoing “Ima Service” every second, while the primary command `nginx` runs. If nginx exits, then the BLINK service will automatically be stopped.

    As an added benefit, any zombie processes left over by nginx will be automatically cleaned up.

    You can also tail log files such as /var/log/nginx/error.log to stderr, edit nginx’s configuration prior to startup and much more

  • For anyone who encounters this, it’s actually not general cloud-init problem, it’s specific to Docker. Docker’s systemd storage service had a dependency on the whole of cloud-init completing (i.e. “After=cloud-final.service”) – see https://github.com/projectatomic/docker-storage-setup/issues/77. This was fixed in October 2016 so as long as apt-get is updated (“package_update: true”) it should be OK.

Leave a Reply