Instancetypes and preferences

FEATURE STATE:

  • v1alpha1 (Experimental) as of the v0.56.0 release
  • v1alpha2 (Experimental) as of the v0.58.0 release
  • This version now captures complete VirtualMachine{Instancetype,ClusterInstnacetype,Preference,ClusterPreference} objects within the created ControllerRevisions
  • This version is backwardly compatible with v1alpha1, no modifications are required to existing instancetypes, preferences or controllerrevisions.

Introduction

KubeVirt's VirtualMachine API contains many advanced options for tuning the performance of a VM that goes beyond what typical users need to be aware of. Users have previously been unable to simply define the storage/network they want assigned to their VM and then declare in broad terms what quality of resources and kind of performance characteristics they need for their VM.

Instancetypes and preferences provide a way to define a set of resource, performance and other runtime characteristics, allowing users to reuse these definitions across multiple VirtualMachines.

VirtualMachineInstancetype

---
apiVersion: instancetype.kubevirt.io/v1alpha2
kind: VirtualMachineInstancetype
metadata:
  name: example-instancetype
spec:
  cpu:
    guest: 1
  memory:
    guest: 128Mi

KubeVirt provides two Instancetype based CRDs, a cluster wide VirtualMachineClusterInstancetype and a namespaced VirtualMachineInstancetype. These CRDs encapsulate the following resource related characteristics of a VirtualMachine through a shared VirtualMachineInstancetypeSpec:

  • CPU : Required number of vCPUs presented to the guest
  • Memory : Required amount of memory presented to the guest
  • GPUs : Optional list of vGPUs to passthrough
  • HostDevices : Optional list of HostDevices to passthrough
  • IOThreadsPolicy : Optional IOThreadsPolicy to be used
  • LaunchSecurity: Optional LaunchSecurity to be used

Anything provided within an instancetype cannot be overridden within the VirtualMachine. For example as CPU and Memory are both required attributes of an instancetype if a user makes any requests for CPU or Memory resources within the underlying VirtualMachine the instancetype will conflict and the request will be rejected during creation.

VirtualMachinePreference

---
apiVersion: instancetype.kubevirt.io/v1alpha2
kind: VirtualMachinePreference
metadata:
  name: example-preference
spec:
  devices:
    preferredDiskBus: virtio
    preferredInterfaceModel: virtio

KubeVirt also provides two further preference based CRDs, again a cluster wide VirtualMachineClusterPreference and namespaced VirtualMachinePreference. These CRDs encapsulate the preferred value of any remaining attributes of a VirtualMachine required to run a given workload, again this is through a shared VirtualMachinePreferenceSpec.

Unlike instancetypes preferences only represent the preferred values and as such can be overridden by values in the VirtualMachine provided by the user.

For example as shown below, if a user has provided a VirtualMachine with a disk bus already defined within a DiskTarget and has also selected a set of preferences with DevicePreference and preferredDiskBus defined the users original choice within the VirtualMachine and DiskTarget are used:

$ cat << EOF | kubectl apply -f - 
---
apiVersion: instancetype.kubevirt.io/v1alpha2
kind: VirtualMachinePreference
metadata:
  name: example-preference-disk-virtio
spec:
  devices:
    preferredDiskBus: virtio
---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: example-preference-user-override
spec:
  preference:
    kind: VirtualMachinePreference
    name: example-preference-disk-virtio
  running: false
  template:
    spec:
      domain:
        memory:
          guest: 128Mi
        devices:
          disks:
          - disk:
              bus: sata
            name: containerdisk
          - disk: {}
            name: cloudinitdisk
        resources: {}
      terminationGracePeriodSeconds: 0
      volumes:
      - containerDisk:
          image: registry:5000/kubevirt/cirros-container-disk-demo:devel
        name: containerdisk
      - cloudInitNoCloud:
          userData: |
            #!/bin/sh

            echo 'printed from cloud-init userdata'
        name: cloudinitdisk
EOF
virtualmachinepreference.instancetype.kubevirt.io/example-preference-disk-virtio created
virtualmachine.kubevirt.io/example-preference-user-override configured


$ virtctl start example-preference-user-override
VM example-preference-user-override was scheduled to start

# We can see the original request from the user within the VirtualMachine lists `containerdisk` with a `SATA` bus
$ kubectl get vms/example-preference-user-override -o json | jq .spec.template.spec.domain.devices.disks
[
  {
    "disk": {
      "bus": "sata"
    },
    "name": "containerdisk"
  },
  {
    "disk": {},
    "name": "cloudinitdisk"
  }
]

# This is still the case in the VirtualMachineInstance with the remaining disk using the `preferredDiskBus` from the preference of `virtio`
$ kubectl get vmis/example-preference-user-override -o json | jq .spec.domain.devices.disks
[
  {
    "disk": {
      "bus": "sata"
    },
    "name": "containerdisk"
  },
  {
    "disk": {
      "bus": "virtio"
    },
    "name": "cloudinitdisk"
  }
]

VirtualMachine

---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: example-vm
spec:
  instancetype:
    kind: VirtualMachineInstancetype
    name: example-instancetype
  preference:
    kind: VirtualMachinePreference
    name: example-preference

The previous instancetype and preference CRDs are matched to a given VirtualMachine through the use of a matcher. Each matcher consists of the following:

Versioning

Versioning of these resources is required to ensure the eventual VirtualMachineInstance created when starting a VirtualMachine does not change between restarts if any referenced instancetype or set of preferences are updated during the lifetime of the VirtualMachine.

This is currently achieved by using ControllerRevision to retain a copy of the VirtualMachineInstancetype or VirtualMachinePreference at the time the VirtualMachine is first started. A reference to these ControllerRevisions are then retained in the VirtualMachineInstancetypeMatcher and VirtualMachinePreferenceMatcher within the VirtualMachine for future use.

$ kubectl.sh apply -f examples/csmall.yaml -f examples/vm-cirros-csmall.yaml
virtualmachineinstancetype.instancetype.kubevirt.io/csmall created
virtualmachine.kubevirt.io/vm-cirros-csmall created

$ kubectl get vm/vm-cirros-csmall -o json | jq .spec.instancetype
{
  "kind": "VirtualMachineInstancetype",
  "name": "csmall",
}

$ virtctl start vm-cirros-csmall
VM vm-cirros-csmall was scheduled to start

$ kubectl get vm/vm-cirros-csmall -o json | jq .spec.instancetype
{
  "kind": "VirtualMachineInstancetype",
  "name": "csmall",
  "revisionName": "vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1"
}

$ kubectl get controllerrevision/vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1 -o json | jq .
{
  "apiVersion": "apps/v1",
  "data": {
    "apiVersion": "instancetype.kubevirt.io/v1alpha2",
    "kind": "VirtualMachineInstancetype",
    "metadata": {
      "creationTimestamp": "2022-09-30T12:20:19Z",
      "generation": 1,
      "name": "csmall",
      "namespace": "default",
      "resourceVersion": "10303",
      "uid": "72c3a35b-6e18-487d-bebf-f73c7d4f4a40"
    },
    "spec": {
      "cpu": {
        "guest": 1
      },
      "memory": {
        "guest": "128Mi"
      }
    }
  },
  "kind": "ControllerRevision",
  "metadata": {
    "creationTimestamp": "2022-09-30T12:20:19Z",
    "name": "vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1",
    "namespace": "default",
    "ownerReferences": [
      {
        "apiVersion": "kubevirt.io/v1",
        "blockOwnerDeletion": true,
        "controller": true,
        "kind": "VirtualMachine",
        "name": "vm-cirros-csmall",
        "uid": "5216527a-1d31-4637-ad3a-b640cb9949a2"
      }
    ],
    "resourceVersion": "10307",
    "uid": "a7bc784b-4cea-45d7-8432-15418e1dd7d3"
  },
  "revision": 0
}


$ kubectl delete vm/vm-cirros-csmall
virtualmachine.kubevirt.io "vm-cirros-csmall" deleted

$ kubectl get controllerrevision/controllerrevision/vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1
Error from server (NotFound): controllerrevisions.apps "vm-cirros-csmall-csmall-72c3a35b-6e18-487d-bebf-f73c7d4f4a40-1" not found

Examples

Various examples are available within the kubevirt repo under /examples. The following uses an example VirtualMachine provided by the containerdisk/fedora repo and replaces much of the DomainSpec with the equivalent instancetype and preferences:

cat << EOF | kubectl apply -f - 
---
apiVersion: instancetype.kubevirt.io/v1alpha2
kind: VirtualMachineInstancetype
metadata:
  name: cmedium
spec:
  cpu:
    guest: 1
  memory:
    guest: 1Gi
---
apiVersion: instancetype.kubevirt.io/v1alpha2
kind: VirtualMachinePreference
metadata:
  name: fedora
spec:
  devices:
    preferredDiskBus: virtio
    preferredInterfaceModel: virtio
    preferredRng: {}
  features:
    preferredAcpi: {}
    preferredSmm: {}
  firmware:
    preferredUseEfi: true
    preferredUseSecureBoot: true    
---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  creationTimestamp: null
  name: fedora
spec:
  instancetype:
    name: cmedium
    kind: virtualMachineInstancetype
  preference:
    name: fedora
    kind: virtualMachinePreference
  runStrategy: Always
  template:
    metadata:
      creationTimestamp: null
    spec:
      domain:
        devices: {}
      volumes:
      - containerDisk:
          image: quay.io/containerdisks/fedora:latest
        name: containerdisk
      - cloudInitNoCloud:
          userData: |-
            #cloud-config
            users:
              - name: admin
                sudo: ALL=(ALL) NOPASSWD:ALL
                ssh_authorized_keys:
                  - ssh-rsa AAAA...
        name: cloudinit
EOF