Libvirt

This provisioner allows Stretchy to run jobs on QEMU virtual machines that can either be emulated or virtualized with KVM. Virtual machines are created on demand by cloning another virtual machine that acts as an image.

Identifier

LIBVIRT

Status

Preview

Since

0.1.0

Maintained by

Stretchy developers

Supported Platforms

Host:

  • Linux

Guest:

  • FreeBSD[1]

  • Linux

  • macOS

  • Windows

Nested virtualization is possible if supported by the host’s hardware.

Hardware Requirements

The libvirt provisioner does not require additional resources beyond the minimum requirements of Stretchy Node Agent.

Images are cloned using copy-on-write. That means you need enough free space to store the virtual machine images to be cloned plus whatever additional data is written to disk by the jobs.

We recommend storing the libvirt storage pools on SSDs that are geared to balanced or write-intensive workloads.

Software Requirements

Host

Guests

Host Configuration

virsh and virtomate must be on the search path of the user stretchy-agent is running as. Furthermore, the same user must have passwordless access to libvirt’s system-mode daemon qemu:///system.

Passwordless access to libvirt’s system-mode daemon qemu:///system can usually be obtained by adding the user that stretchy-agent is running as to the group libvirt. Please consult the documentation of your operating system on how to do so.

To validate whether virsh is working correctly, run the following command:

virsh -c qemu:///system list --all

For virtomate, run:

virtomate -p -c qemu:///system domain-list

Both commands should list the virtual machines that you want to use with Stretchy.

Agent Configuration

Apart from setting the provisioner identifier (LIBVIRT) and the provisioner parallelism (start with 3), no further configuration is necessary.

The parallelism value depends on the number of machines the provisioner should start and stop simultaneously. The longer a machines needs to boot, the longer the provisioner has to wait and cannot work on another machine. If the number of running machines is consistently lower than the number of machines the node could host, increase the parallelism gradually.

Image Configuration

Install the guest operating system as you normally would, for example, with the help of Virtual Machine Manager or packer. All job sources expect a local user named stretchy to be present with a home directory according to Table 1.

Table 1. Expected home directories by system
System family Home directory

BSD

/home/stretchy

Linux

/home/stretchy

macOS

/Users/stretchy

Windows

C:\Users\stretchy
For cloning to succeed, virtomate requires all storage volumes including NVRAM flash to be stored in a libvirt storage pool. See the manpage of virtomate domain-clone for details.
Some Linux distributions, notably Ubuntu, use /etc/machine-id to request an IP address using DHCP. Ensure that each guest has a unique /etc/machine-id and, therefore, IP address.
We highly recommend employing cloud-init to set the virtual machine’s hostname, to replace its SSH host key, and to change any passwords when the guest boots for the first time. On Linux, systemd-firstboot might be an option, too.
Automatic updates can slow down or even interrupt the execution of jobs. Therefore, we recommend disabling automatic updates on all guest systems.

Then, install all required software, including your job source’s agent.

On Windows, it is necessary to change the Log on user of QEMU Guest Agent to the user stretchy, for example, by running sc config QEMU-GA obj=.\stretchy password=<password> in an elevated command prompt. Otherwise, jobs will be run as NT Authority\System.

If jobs should be able to access a graphical user interface, for example, to perform GUI testing, automatic login must be enabled on macOS and Windows for the user stretchy. On macOS, the job runner will be started in such a way that it has access to an Aqua session.

To verify that the QEMU Guest Agent works correctly on a guest, start the virtual machine and run the following command after replacing <domain> with the name of your virtual machine:

virtomate -p -c qemu:///system guest-run <domain> -- echo "Hello"

The output should be as follows:

{
  "exit_code": 0,
  "signal": null,
  "stderr": null,
  "stderr_truncated": false,
  "stdout": "Hello\n",
  "stdout_truncated": false
}

For Windows guests, use:

virtomate -p -c qemu:///system guest-run <domain> -- powershell.exe -Command "Write-Host 'Hello'"

Forgejo Runner

Download and extract Forgejo Runner. Put it in the expected place as outlined by Table 2. No further installation or configuration is necessary. Stretchy will take care of it for you.

It is recommended to use the latest version of Forgejo Runner. The oldest supported version is Forgejo Runner 12.8.
Table 2. Programs invoked to start Forgejo Runner
System family Program

BSD

/usr/local/bin/forgejo-runner

Linux

/usr/local/bin/forgejo-runner

macOS

/usr/local/bin/forgejo-runner

Windows

C:\Users\stretchy\forgejo-runner.exe

GitHub Actions Runner

After installing all dependencies, download and extract the GitHub Actions Runner as explained in its README. Any further configuration or a registration on GitHub is not necessary. Stretchy will do that automatically for you.

The provisioner will start GitHub Actions Runner by invoking the programs listed in Table 3.

Table 3. Programs invoked to start GitHub Actions Runner
System family Directory

Linux

/home/stretchy/actions-runner/run.sh

macOS

/Users/stretchy/actions-runner/run.sh

Windows

C:\actions-runner\run.cmd
GitHub Actions Runner does not support BSD.

Jenkins

After installing a suitable version of Java, download Jenkins' agent.jar from your Jenkins instance (for example, https://jenkins.example.com/jnlpJars/agent.jar) and place it in the home directory of the local user stretchy. The provisioner expects it at the paths listed in Table 4.

Table 4. Programs invoked to start Jenkins Agent
System family Directory

BSD

/home/stretchy/agent.jar

Linux

/home/stretchy/agent.jar

macOS

/Users/stretchy/agent.jar

Windows

C:\Users\stretchy\agent.jar

Image Registration

To register an image in Stretchy Orchestrator, enter the name of the corresponding libvirt domain as listed by virsh -c qemu:///system list --all into the field Name and LIBVIRT into the field Provisioner.

Do not prefix the name of your libvirt domains with stretchy-. The libvirt provisioner treats domains prefixed with stretchy- like stretchy-ubuntu as managed domains (domains dynamically created by the provisioner). It might either complain about them or even delete them without asking.
1. Please share your experience with other BSDs with us.