Hotplug Network Interfaces¶
Release:
- v1.1.0: Alpha
- v1.3.0: Beta
- v1.4.0: GA
KubeVirt supports hotplugging and unplugging network interfaces into a running Virtual Machine (VM).
Hotplug is supported for interfaces using the virtio model connected through
bridge binding
or SR-IOV binding.
Hot-unplug is supported only for interfaces connected through bridge binding.
Requirements¶
Adding an interface to a KubeVirt Virtual Machine requires first an interface to be added to a running pod. This is not trivial, and has some requirements:
- Multus Dynamic Networks Controller: this daemon will listen to annotation changes, and trigger Multus to configure a new attachment for this pod.
- Multus CNI running as a thick plugin: this Multus version exposes an endpoint to create attachments for a given pod on demand.
Note: For older Kubevirt versions (from v1.1 until v1.3), the
HotplugNICsfeature-gate) must be enabled. From Kubevirt v1.4, the FG is not needed and should be removed if set.
Adding an interface to a running VM¶
First start a VM. You can refer to the following example:
---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: vm-fedora
spec:
runStrategy: Always
template:
spec:
domain:
devices:
disks:
- disk:
bus: virtio
name: containerdisk
interfaces:
- masquerade: {}
name: defaultnetwork
rng: {}
resources:
requests:
memory: 1024M
networks:
- name: defaultnetwork
pod: {}
terminationGracePeriodSeconds: 0
volumes:
- containerDisk:
image: quay.io/kubevirt/fedora-with-test-tooling-container-disk:devel
name: containerdisk
You should configure a network attachment definition - where the pod interface configuration is held. The snippet below shows an example of a very simple one:
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: new-fancy-net
spec:
config: '{
"cniVersion": "0.3.1",
"type": "bridge",
"mtu": 1300,
"name":"new-fancy-net"
}'
Once the virtual machine is running, and the attachment configuration provisioned, the user can request the interface hotplug operation by editing the VM spec template and adding the desired interface and network:
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: vm-fedora
template:
spec:
domain:
devices:
interfaces:
- name: defaultnetwork
masquerade: {}
# new interface
- name: dyniface1
bridge: {}
networks:
- name: defaultnetwork
pod: {}
# new network
- name: dyniface1
multus:
networkName: new-fancy-net
...
Note:
virtctladdinterfaceandremoveinterfacecommands are no longer available, hotplug/unplug interfaces is done by editing the VM spec template.
The interface and network will be added to the corresponding VMI object as well by Kubevirt.
You can now check the VMI status for the presence of this new interface:
Removing an interface from a running VM¶
Following the example above, the user can request an interface unplug operation
by editing the VM spec template and set the desired interface state to absent:
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: vm-fedora
template:
spec:
domain:
devices:
interfaces:
- name: defaultnetwork
masquerade: {}
# set the interface state to absent
- name: dyniface1
state: absent
bridge: {}
networks:
- name: defaultnetwork
pod: {}
- name: dyniface1
multus:
networkName: new-fancy-net
Note: Existing VMs from version v0.59.0 and below do not support hot-unplug interfaces.
Migration based hotplug¶
In case your cluster doesn't run Multus as thick plugin and Multus Dynamic Networks controller, it's possible to hotplug an interface by migrating the VM.
The actual attachment won't take place immediately, and the new interface will be available in the guest once the migration is completed.
Add new interface¶
Add the desired interface and network to the VM spec template:
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: vm-fedora
template:
spec:
domain:
devices:
interfaces:
- name: defaultnetwork
masquerade: {}
# new interface
- name: dyniface1
bridge: {}
networks:
- name: defaultnetwork
pod: {}
# new network
- name: dyniface1
multus:
networkName: new-fancy-net
...
At this point the interface and network will be added to the corresponding VMI object as well, but won't be attached to the guest.
Migrate the VM¶
cat <<EOF kubectl apply -f -
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstanceMigration
metadata:
name: migration-job
spec:
vmiName: vmi-fedora
EOF
Once the migration is completed the VM will have the new interface attached.
Note: It is recommended to avoid performing migrations in parallel to a hotplug operation. It is safer to assure hotplug succeeded or at least reached the VMI specification before issuing a migration.
Remove interface¶
Set the desired interface state to absent in the VM spec template:
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: vm-fedora
template:
spec:
domain:
devices:
interfaces:
- name: defaultnetwork
masquerade: {}
# set the interface state to absent
- name: dyniface1
state: absent
bridge: {}
networks:
- name: defaultnetwork
pod: {}
- name: dyniface1
multus:
networkName: new-fancy-net
At this point the subject interface should be detached from the guest but exist in the pod.
Note: Existing VMs from version v0.59.0 and below do not support hot-unplug interfaces.
Migrate the VM¶
cat <<EOF kubectl apply -f -
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstanceMigration
metadata:
name: migration-job
spec:
vmiName: vmi-fedora
EOF
Once the VM is migrated, the interface will not exist in the migration target pod.
Note: It is recommended to avoid performing migrations in parallel to an unplug operation. It is safer to assure unplug succeeded or at least reached the VMI specification before issuing a migration.
SR-IOV interfaces¶
Kubevirt supports hot-plugging of SR-IOV interfaces to running VMs.
Similar to bridge binding interfaces, edit the VM spec template and add the desired SR-IOV interface and network:
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: vm-fedora
template:
spec:
domain:
devices:
interfaces:
- name: defaultnetwork
masquerade: {}
# new interface
- name: sriov-net
sriov: {}
networks:
- name: defaultnetwork
pod: {}
# new network
- name: sriov-net
multus:
networkName: sriov-net-1
...
At this point the interface and network will be added to the corresponding VMI object as well, but won't be attached to the guest.
Migrate the VM¶
cat <<EOF kubectl apply -f -
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstanceMigration
metadata:
name: migration-job
spec:
vmiName: vmi-fedora
EOF
Once the VM is migrated, the interface will not exist in the migration target pod. Due to limitation of Kubernetes device plugin API to allocate resources dynamically, the SR-IOV device plugin cannot allocate additional SR-IOV resources for Kubevirt to hotplug. Thus, SR-IOV interface hotplug is limited to migration based hotplug only, regardless of Multus "thick" version.
Virtio Limitations¶
The hotplugged interfaces have model: virtio. This imposes several
limitations: each interface will consume a PCI slot in the VM, and there are a
total maximum of 32. Furthermore, other devices will also use these PCI slots
(e.g. disks, guest-agent, etc).
Kubevirt reserves resources for 4 interface to allow later hotplug operations. The actual maximum amount of available resources depends on the machine type (e.g. q35 adds another PCI slot). For more information on maximum limits, see libvirt documentation.
Yet, upon a VM restart, the hotplugged interface will become part of the standard networks; this mitigates the maximum hotplug interfaces (per machine type) limitation.
Note: The user can execute this command against a stopped VM - i.e. a VM without an associated VMI. When this happens, KubeVirt mutates the VM spec template on behalf of the user.