Christoffer Dall lead a session today at Linaro Connect discussing standards for portable ARM virtual machines (video). About a week ago, Christoffer posted a draft specification to the linux-arm-kernel, kvm and xen mailing lists which attracted lots of useful feedback. Today we went over the major points of issue and Christopher is going to take the feedback to prepare a new draft.

Many of the issues raised boil down to how much reach the spec should have. If it specifies too much, then it will be burdensome for vendors to be compliant, but if it specifies too little then it won’t be useful for making portable disk images. Today we talked about how specific it must be on the topics of required hardware, required virtual interfaces (virtio, xenbus), firmware interface (UEFI) and hardware description (ACPI, FDT).

We also talked about the use-cases covered by this spec. For instance, while there is interest in supporting some hypothetical future version of ARM Windows as either a host or a guest, it is pointless to try and guess what requirements Microsoft will have. For now the focus is on Linux hosts running either Xen, KVM or QEMU, with guests running predominantly Linux (while still supporting any guest OS that conforms). OS vendors should be able to use the spec to design installation and update tools that will work with any compliant virtual machine.

The ARM Server Base System Architecture (SBSA) specification defines the basic requirements for ARM server hardware. Christoffer used the SBSA as a starting point, but quickly realized that the peripheral options described in the SBSA makes little sense in a virtual environment. For instance, a virtual machine can certainly emulate a SATA controller, but it can provide far better performance with an interface designed for virtualization. It was asked if the spec should specify a choice of either virtio or xenbus, but the problem with doing so is it effectively requires OSes to implement support for both in order to be compliant. This isn’t a problem for Linux guests because the kernel already has drivers for both, but it could be a problem for non-Linux guests.

Instead the choice was made to treat virtual buses in exactly the same way we treat real hardware; it is still up to the OS to include driver support for the platform it is running on. OS vendors are strongly encouraged to support both, but the spec does not require them to do so. If only one is supported then the onus is on them to list it in their own requirements.

Particular attention was given to the SBSA serial port requirement. Level 1 of the SBSA requires the platform implement a debug port which is register compatible with ARM’s pl011 UART. Ian Campbell and Stefano Stabellini from Citrix were concerned that implementing full pl011 emulation would perform poorly and would be require a lot of work to implement. However, Alexander Graf pointed out that an always available console device would eliminate a lot of the pain of failed booting without any log output. It was also pointed out that the SBSA does not actually require a full pl011 implementation. DMA and IRQ support are not necessary, which makes emulation trivial, and the virtual UART is only expected to be used during early boot scenarios. Normally console output will be reported first via the UEFI console before ExitBootServices() is called, and then via the VM’s preferred console device. At the close of the discussion we decided to require the SBSA debug port definition in the VM spec.

The requirement of UEFI for the firmware interface was mostly uncontroversial. In the earlier mailing list discussion, Dennis Gilmore did take issue with specifying UEFI over U-Boot given that UEFI is not in heavy use on 32-bit ARM. U-Boot is also making strides forward in standardizing the boot flow which would make it it more suitable for VM scenarios. Dennis is concerned that UEFI would require a lot of new effort to get working. However, that work has already been completed. There is a 32-bit port of UEFI running under QEMU, mainline GRUB includes ARM UEFI support, and merging kernel support is in progress.

None of the VM developers in the room today seemed concerned about requiring UEFI for virtual firmware, and the UEFI spec covers quite a few standard booting scenarios including, removable media, network booting, and booting from a block device. The feeling is that it is important for both 64-bit and 32-bit virtual machines to have the same behaviour and so the UEFI requirement will remain.

Deciding whether an FDT or an ACPI hardware description is required was more of an concern. Jon Masters from Red Hat has previously stated that Red Hat Enterprise Linux will only support booting with ACPI. There is concern that the specification will not be acceptable to Red Hat if it does not require ACPI. However, ACPI is still a work in progress and we don’t yet know how to implement it in a VM. Since all of the VMs already use FDT, and will continue to do so for the foreseeable future, it was decided to make FDT support mandatory in version 1 of the spec. A future version 2 will allow ACPI to be provided in addition to FDT with the expectation that an OS vendor can choose to make ACPI support mandatory for their product.

For the next steps, Christoffer is going to take all the comments from the mailing list and today’s meeting and he will post a second draft of the spec. Then after further feedback, the specification will probably get published, possibly as a Linaro whitepaper.

Follow