Virtual hardware

Fine-tuning different aspects of the hardware which are not device related (BIOS, mainboard, etc.) is sometimes necessary to allow guest operating systems to properly boot and reboot.

Machine Type

QEMU is able to work with two different classes of chipsets for x86_64, so called machine types. The x86_64 chipsets are i440fx (also called pc) and q35. They are versioned based on qemu-system-${ARCH}, following the format pc-${machine_type}-${qemu_version}, e.g.pc-i440fx-2.10 and pc-q35-2.10.

KubeVirt defaults to QEMU's newest q35 machine type. If a custom machine type is desired, it is configurable through the following structure:

metadata:
  name: myvmi
spec:
  domain:
    machine:
      # This value indicates QEMU machine type.
      type: pc-q35-2.10
    resources:
      requests:
        memory: 512M
    devices:
      disks:
      - name: myimage
        disk: {}
  volumes:
    - name: myimage
      persistentVolumeClaim:
        claimName: myclaim

Comparison of the machine types' internals can be found at QEMU wiki.

BIOS/UEFI

All virtual machines use BIOS by default for booting.

It is possible to utilize UEFI/OVMF by setting a value via spec.firmware.bootloader:

apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
  labels:
    special: vmi-alpine-efi
  name: vmi-alpine-efi
spec:
  domain:
    devices:
      disks:
      - disk:
          bus: virtio
        name: containerdisk
    features:
      smm:
        enabled: true
    firmware:
      # this sets the bootloader type
      bootloader:
        efi: {}

Enabling EFI automatically enables Secure Boot, unless the secureBoot field under efi is set to false.
Secure Boot itself requires the SMM CPU feature to be enabled as above, which does not happen automatically, for security reasons.

SMBIOS Firmware

In order to provide a consistent view on the virtualized hardware for the guest OS, the SMBIOS UUID can be set to a constant value via spec.firmware.uuid:

metadata:
  name: myvmi
spec:
  domain:
    firmware:
      # this sets the UUID
      uuid: 5d307ca9-b3ef-428c-8861-06e72d69f223
      serial: e4686d2c-6e8d-4335-b8fd-81bee22f4815
    resources:
      requests:
        memory: 512M
    devices:
      disks:
      - name: myimage
        disk: {}
  volumes:
    - name: myimage
      persistentVolumeClaim:
        claimName: myclaim

In addition, the SMBIOS serial number can be set to a constant value via spec.firmware.serial, as demonstrated above.

CPU

Note: This is not related to scheduling decisions or resource assignment.

Topology

Setting the number of CPU cores is possible via spec.domain.cpu.cores. The following VM will have a CPU with 3 cores:

metadata:
  name: myvmi
spec:
  domain:
    cpu:
      # this sets the cores
      cores: 3
    resources:
      requests:
        memory: 512M
    devices:
      disks:
      - name: myimage
        disk: {}
  volumes:
    - name: myimage
      persistentVolumeClaim:
        claimName: myclaim

Labeling nodes with cpu models, cpu features and machine types

KubeVirt can create node selectors based on VM cpu models, features, and machine types. With these node selectors, VMs will be scheduled on the nodes that support the matching VM cpu model, features, and machine type.

To properly label the node, user can use Kubevirt Node-labeller, which creates all necessary labels or create node labels by himself.

Kubevirt node-labeller creates 4 types of labels: cpu models, cpu features, kvm info and machine types. It uses libvirt to get all supported cpu models and cpu features on host and then Node-labeller creates labels from cpu models and machine types.

Node-labeller supports obsolete list of cpu models and minimal baseline cpu model for features. Both features can be set via KubeVirt CR:

apiVersion: kubevirt.io/v1
kind: Kubevirt
metadata:
  name: kubevirt
  namespace: kubevirt
spec:
  ...
  configuration:
    obsoleteCPUModels:
      486: true
      pentium: true
...

Obsolete cpus will not be inserted in labels. If KubeVirt CR doesn't contain obsoleteCPUModels variable, Labeller sets default values ("pentium, pentium2, pentium3, pentiumpro, coreduo, n270, core2duo, Conroe, athlon, phenom, kvm32, kvm64, qemu32 and qemu64").

User can change obsoleteCPUModels by adding / removing cpu model in config map. Kubevirt then update nodes with new labels.

For homogenous cluster / clusters without live migration enabled it's possible to disable the node labeler and avoid adding labels to the nodes by adding the following annotation to the nodes:

node-labeller.kubevirt.io/skip-node.

Model

Note: If CPU model wasn't defined, the VM will have CPU model closest to one that used on the node where the VM is running.

Note: CPU model is case sensitive.

Setting the CPU model is possible via spec.domain.cpu.model. The following VM will have a CPU with the Conroe model:

apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
  name: myvmi
spec:
  domain:
    cpu:
      # this sets the CPU model
      model: Conroe
...

You can check list of available models here.

When CPUNodeDiscovery feature-gate is enabled and VM has cpu model, Kubevirt creates node selector with format: cpu-model.node.kubevirt.io/<cpuModel>, e.g. cpu-model.node.kubevirt.io/Conroe. When VM doesn’t have cpu model, then no node selector is created.

Enabling default cluster cpu model

To enable the default cpu model, user may add the cpuModel field in the KubeVirt CR.

apiVersion: kubevirt.io/v1
kind: KubeVirt
metadata:
  name: kubevirt
  namespace: kubevirt
spec:
  ...
  configuration:
    cpuModel: "EPYC"
...

Default CPU model is set when vmi doesn't have any cpu model. When vmi has cpu model set, then vmi's cpu model is preferred. When default cpu model is not set and vmi's cpu model is not set too, host-model will be set. Default cpu model can be changed when kubevirt is running. When CPUNodeDiscovery feature gate is enabled Kubevirt creates node selector with default cpu model.

CPU model special cases

As special cases you can set spec.domain.cpu.model equals to: - host-passthrough to passthrough CPU from the node to the VM

metadata:
  name: myvmi
spec:
  domain:
    cpu:
      # this passthrough the node CPU to the VM
      model: host-passthrough
...

• host-model to get CPU on the VM close to the node one

metadata:
  name: myvmi
spec:
  domain:
    cpu:
      # this set the VM CPU close to the node one
      model: host-model
...

See the CPU API reference for more details.

Features

Setting CPU features is possible via spec.domain.cpu.features and can contain zero or more CPU features :

metadata:
  name: myvmi
spec:
  domain:
    cpu:
      # this sets the CPU features
      features:
      # this is the feature's name
      - name: "apic"
      # this is the feature's policy
       policy: "require"
...

Note: Policy attribute can either be omitted or contain one of the following policies: force, require, optional, disable, forbid.

Note: In case a policy is omitted for a feature, it will default to require.

Behaviour according to Policies:

• All policies will be passed to libvirt during virtual machine creation.
• In case the feature gate "CPUNodeDiscovery" is enabled and the policy is omitted or has "require" value, then the virtual machine could be scheduled only on nodes that support this feature.
• In case the feature gate "CPUNodeDiscovery" is enabled and the policy has "forbid" value, then the virtual machine would not be scheduled on nodes that support this feature.

Full description about features and policies can be found here.

When CPUNodeDiscovery feature-gate is enabled Kubevirt creates node selector from cpu features with format: cpu-feature.node.kubevirt.io/<cpuFeature>, e.g. cpu-feature.node.kubevirt.io/apic. When VM doesn’t have cpu feature, then no node selector is created.

Clock

Guest time

Sets the virtualized hardware clock inside the VM to a specific time. Available options are

• utc

• timezone

See the Clock API Reference for all possible configuration options.

utc

If utc is specified, the VM's clock will be set to UTC.

metadata:
  name: myvmi
spec:
  domain:
    clock:
      utc: {}
    resources:
      requests:
        memory: 512M
    devices:
      disks:
      - name: myimage
        disk: {}
  volumes:
    - name: myimage
      persistentVolumeClaim:
        claimName: myclaim

timezone

If timezone is specified, the VM's clock will be set to the specified local time.

metadata:
  name: myvmi
spec:
  domain:
    clock:
      timezone: "America/New York"
    resources:
      requests:
        memory: 512M
    devices:
      disks:
      - name: myimage
        disk: {}
  volumes:
    - name: myimage
      persistentVolumeClaim:
        claimName: myclaim

Timers

• pit

• rtc

• kvm

• hyperv

A pretty common timer configuration for VMs looks like this:

metadata:
  name: myvmi
spec:
  domain:
    clock:
      utc: {}
      # here are the timer
      timer:
        hpet:
          present: false
        pit:
          tickPolicy: delay
        rtc:
          tickPolicy: catchup
        hyperv: {}
    resources:
      requests:
        memory: 512M
    devices:
      disks:
      - name: myimage
        disk: {}
  volumes:
    - name: myimage
      persistentVolumeClaim:
        claimName: myclaim

hpet is disabled,pit and rtc are configured to use a specific tickPolicy. Finally, hyperv is made available too.

See the Timer API Reference for all possible configuration options.

Note: Timer can be part of a machine type. Thus it may be necessary to explicitly disable them. We may in the future decide to add them via cluster-level defaulting, if they are part of a QEMU machine definition.

Random number generator (RNG)

You may want to use entropy collected by your cluster nodes inside your guest. KubeVirt allows to add a virtio RNG device to a virtual machine as follows.

metadata:
  name: vmi-with-rng
spec:
  domain:
    devices:
      rng: {}

For Linux guests, the virtio-rng kernel module should be loaded early in the boot process to acquire access to the entropy source. Other systems may require similar adjustments to work with the virtio RNG device.

Note: Some guest operating systems or user payloads may require the RNG device with enough entropy and may fail to boot without it. For example, fresh Fedora images with newer kernels (4.16.4+) may require the virtio RNG device to be present to boot to login.

Video and Graphics Device

By default a minimal Video and Graphics device configuration will be applied to the VirtualMachineInstance. The video device is vga compatible and comes with a memory size of 16 MB. This device allows connecting to the OS via vnc.

It is possible to not attach the video device by setting spec.domain.devices.autoattachGraphicsDevice to false:

metadata:
  name: myvmi
spec:
  domain:
    devices:
      autoattachGraphicsDevice: false
      disks:
      - name: myimage
        disk: {}
  volumes:
    - name: myimage
      persistentVolumeClaim:
        claimName: myclaim

VMIs without graphics and video devices are very often referenced as headless VMIs.

If using a huge amount of small VMs this can be helpful to increase the VMI density per node, since no memory needs to be reserved for video.

Custom Video Device Types

With the VideoConfig feature gate enabled, it is possible to configure custom video device types to override the default video configuration. This allows you to specify video device types such as virtio, vga, or bochs depending on your guest operating system requirements and performance needs.

Note: As of KubeVirt 1.6, the VideoConfig feature gate is currently in alpha status and is disabled by default. As an alpha feature: - It must be explicitly enabled to use custom video device configuration

Enabling VideoConfig Feature Gate

To enable custom video device configuration, the VideoConfig feature gate must be activated in the KubeVirt CR.

For HCO users: As of version 1.15.2, HCO support for VideoConfig feature gate is not yet available. It will be supported in a future release.

For direct KubeVirt configuration:

kubectl patch kubevirt kubevirt -n kubevirt --type=json \
  -p='[{"op": "add", "path": "/spec/configuration/developerConfiguration/featureGates/-", "value": "VideoConfig"}]'

Alternatively, you can edit the kubevirt CR directly:

kubectl edit kubevirt kubevirt -n kubevirt

spec:
  configuration:
    developerConfiguration:
      featureGates:
      - "VideoConfig"

Configuring Video Device Types

Once the feature gate is enabled, you can specify the video device type in the VM template specification under spec.template.spec.domain.devices.video. For example:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: my-vm-with-virtio-gpu
spec:
  template:
    spec:
      domain:
        devices:
          video:
            type: virtio
        resources:
          requests:
            memory: 2Gi

Supported video device types:

• virtio: Provides better performance and is recommended for modern guest operating systems for which virtio drivers are available
• vga: Standard VGA-compatible device (default on AMD/x86_64 with BIOS)
• bochs: Similar to vga but more efficient (default on AMD/x86_64 with EFI)
• cirrus: Legacy Cirrus Logic device
• ramfb: Simple framebuffer device, useful for ARM architectures

Architecture-Specific Behavior

The VideoConfig feature gate allows overriding default video device types that vary by architecture:

Architecture	Boot Mode	Default Type	Supported Types
AMD/x86_64	BIOS	vga	virtio, vga, bochs, cirrus, ramfb
AMD/x86_64	EFI	bochs	virtio, vga, bochs, cirrus, ramfb
ARM64	BIOS/EFI	virtio	virtio, ramfb
s390x	BIOS/EFI	virtio	virtio

Benefits of Custom Video Device Configuration

Using custom video device types provides several advantages:

• Performance: virtio video devices offer better performance compared to legacy VGA emulation
• Resolution flexibility: With virtio you can set custom display resolutions that aren't limited by VGA constraints
• Memory efficiency: Some video types like ramfb are more memory efficient for headless or console-only operations

Example for ARM64 with explicit virtio configuration:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: arm64-vm-custom-video
spec:
  template:
    spec:
      domain:
        devices:
          video:
            type: virtio
        resources:
          requests:
            memory: 4Gi

Example for AMD/x86_64 overriding the default VGA video type to virtio:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: amd64-vm-virtio-video
spec:
  template:
    spec:
      domain:
        devices:
          video:
            type: virtio
        resources:
          requests:
            memory: 2Gi

Similarly, you can override to other supported types like bochs, cirrus, or ramfb:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: amd64-vm-bochs-video
spec:
  template:
    spec:
      domain:
        devices:
          video:
            type: bochs
        resources:
          requests:
            memory: 2Gi

Note: When using custom video device types, ensure that your guest operating system has the appropriate drivers installed. For virtio-gpu devices, virtio graphics drivers are required for optimal performance. Bochs also requires a driver, but only for BIOS guests, since our EFI firmware (OVMF) includes a bochs driver.

Features

KubeVirt supports a range of virtualization features which may be tweaked in order to allow non-Linux based operating systems to properly boot. Most noteworthy are

• acpi

• apic

• hyperv

A common feature configuration is shown by the following example:

apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
  name: myvmi
spec:
  domain:
    # typical features
    features:
      acpi: {}
      apic: {}
      hyperv:
        relaxed: {}
        vapic: {}
        spinlocks:
          spinlocks: 8191
    resources:
      requests:
        memory: 512M
    devices:
      disks:
      - name: myimage
        disk: {}
  volumes:
    - name: myimage
      persistentVolumeClaim:
        claimname: myclaim

See the Features API Reference for all available features and configuration options.

Resources Requests and Limits

An optional resource request can be specified by the users to allow the scheduler to make a better decision in finding the most suitable Node to place the VM.

apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
  name: myvmi
spec:
  domain:
    resources:
      requests:
        memory: "1Gi"
        cpu: "1"
      limits:
        memory: "2Gi"
        cpu: "2"
      disks:
      - name: myimage
        disk: {}
  volumes:
    - name: myimage
      persistentVolumeClaim:
        claimname: myclaim

CPU

Specifying CPU limits will determine the amount of cpu shares set on the control group the VM is running in, in other words, the amount of time the VM's CPUs can execute on the assigned resources when there is a competition for CPU resources.

For more information please refer to how Pods with resource limits are run.

Memory Overhead

Various VM resources, such as a video adapter, IOThreads, and supplementary system software, consume additional memory from the Node, beyond the requested memory intended for the guest OS consumption. In order to provide a better estimate for the scheduler, this memory overhead will be calculated and added to the requested memory.

The default memory overhead has been set to 32Mi to account for these additional system resources.

Please see how Pods with resource requests are scheduled for additional information on resource requests and limits.

Hugepages

KubeVirt give you possibility to use hugepages as backing memory for your VM. You will need to provide desired amount of memory resources.requests.memory and size of hugepages to use memory.hugepages.pageSize, for example for x86_64 architecture it can be 2Mi.

apiVersion: kubevirt.io/v1alpha1
kind: VirtualMachine
metadata:
  name: myvm
spec:
  domain:
    resources:
      requests:
        memory: "64Mi"
    memory:
      hugepages:
        pageSize: "2Mi"
    disks:
    - name: myimage
      disk: {}
  volumes:
    - name: myimage
      persistentVolumeClaim:
        claimname: myclaim

In the above example the VM will have 64Mi of memory, but instead of regular memory it will use node hugepages of the size of 2Mi.

Limitations

• a node must have pre-allocated hugepages

• hugepages size cannot be bigger than requested memory

• requested memory must be divisible by hugepages size

• hugepages uses by default memfd. Memfd is supported from kernel >= 4.14. If you run on an older host (e.g centos 7.9), it is required to disable memfd with the annotation kubevirt.io/memfd: "false" in the VMI metadata annotation.

Input Devices

Tablet

Kubevirt supports input devices. The only type which is supported is tablet. Tablet input device supports only virtio and usb bus. Bus can be empty. In that case, usb will be selected.

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: myvm
spec:
  domain:
    devices:
      inputs:
      - type: tablet
        bus: virtio
        name: tablet1
      disks:
      - name: myimage
        disk: {}
  volumes:
    - name: myimage
      persistentVolumeClaim:
        claimname: myclaim