Hotplug Network Interfaces¶
KubeVirt now supports hotplugging network interfaces into a running Virtual
Machine Instance (VMI). Hotplug is only supported for interfaces using the
virtio
model connected through
bridge binding.
Requirements¶
Adding an interface to a KubeVirt Virtual Machine requires first an interface to be added to a running pod. This is not trivial, and has some requirements:
- multus dynamic networks controller: this daemon will listen to annotation changes, and trigger multus to configure a new attachment for this pod.
- multus running as a thick plugin: this multus version exposes an endpoint to create attachments for a given pod on demand.
Enabling network interface hotplug support¶
Network interface hotplug support must be enabled via a
feature gate.
The feature gates array in the KubeVirt CR must feature HotplugNICs
.
Adding an interface to a running VMI¶
First start a VMI. You can refer to the following example:
---
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
name: vmi-fedora
spec:
domain:
devices:
disks:
- disk:
bus: virtio
name: containerdisk
- disk:
bus: virtio
name: cloudinitdisk
interfaces:
- masquerade: {}
name: defaultnetwork
rng: {}
resources:
requests:
memory: 1024M
networks:
- name: defaultnetwork
pod: {}
terminationGracePeriodSeconds: 0
volumes:
- containerDisk:
image: quay.io/kubevirt/fedora-with-test-tooling-container-disk:devel
name: containerdisk
- cloudInitNoCloud:
networkData: |
version: 2
ethernets:
eth0:
dhcp4: true
userData: |-
#!/bin/bash
echo "fedora" |passwd fedora --stdin
name: cloudinitdisk
You should configure a network attachment definition - where the pod interface configuration is held. The snippet below shows an example of a very simple one:
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: new-fancy-net
spec:
config: '{
"cniVersion": "0.3.1",
"type": "bridge",
"mtu": 1300,
"name":"new-fancy-net"
}'
Please refer to the multus documentation for more info.
Once the virtual machine is running, and the attachment configuration provisioned, the user can request the interface hotplug operation. Please refer to the following snippet:
NOTE: You can use the --help
parameter for more information on each
parameter.
You can now check the VMI status for the presence of this new interface:
Migration based hotplug¶
In case your cluster doesn't run Multus as thick plugin and Multus Dynamic Networks controller, it's possible to hotplug an interface by migrating the VM.
The actual attachment won't take place immediately, and the new interface will be available in the guest once the migration is completed.
Add new interface¶
At this point the new interface is added to the spec but will not be attached to the running VM.Migrate the VM¶
cat <<EOF kubectl apply -f -
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstanceMigration
metadata:
name: migration-job
spec:
vmiName: vmi-fedora
EOF
Once the migration is completed the VM will have the new interface attached.
Virtio Limitations¶
The hotplugged interfaces have model: virtio
. This imposes several
limitations: each interface will consume a PCI slot in the VM, and there are a
total maximum of 32. Furthermore, other devices will also use these PCI slots
(e.g. disks, guest-agent, etc).
For the q35
machine type, the issue is even worse: the users will be able to
hotplug a single network interface into the guest. You can find more
information in the
libvirt documentation.
Persisting Hotplugged Interfaces¶
Users may want an hotplugged interface to become part of the standard networks / interfaces after a restart. This use case requires users to template the running VMI from a VM object.
To achieve this use case, users should invoke addinterface
with the
--persist
flag: this will add the interface to the running VMI object, and
also mutate the VM spec template:
virtctl addinterface vmi-fedora --network-attachment-definition-name new-fancy-net --name dyniface1 --persist
Thus, upon a VM restart, the new interface will be made available in the VMI;
this mitigates the single hotplug interface per q35
machine type limitation.
NOTE: the user can execute this command against a stopped VM - i.e. a VM without an associated VMI. When this happens, KubeVirt mutates the VM spec template on behalf of the user.