Federated KubeVirt

Federated KubeVirt is a reference implementation of deploying and managing KubeVirt across multiple Kubernetes clusters using Federation-v2.

Federation-v2 is an API and control-plane for actively managing multiple Kubernetes clusters and applications in those clusters. This makes Federation-v2 a viable solution for managing KubeVirt deployments that span multiple Kubernetes clusters.

Federation-v2 Deployment

We assume federation is already deployed (using latest stable version) and you have configured your two clusters with context cluster1 and cluster2

Federated KubeVirt Deployment

We create KubeVirt namespace on first cluster:

kubectl create ns kubevirt

We then create a placement for this namespace to get replicated to the second cluster.

kubectl create -f federated_namespace.yaml

NOTE: This yaml file was generated for version 0.14.0. but feel free to edit in order to use a more recent version of the operator

We create the federated objects required as per kubevirt deployment:

kubefed2 enable ClusterRoleBinding
kubefed2 enable CustomResourceDefinition

And federated kubevirt itself, with placements so that it gets deployed at both sites.

kubectl create -f federated_kubevirt-operator.yaml

This gets KubeVirt operator deployed at both sites, which creates the Custom Resource definition KubeVirt. We then deploy kubevirt by federating this CRD and creates an instance of it.

kubefed2 enable kubevirts
kubectl create -f federated_kubevirt-cr.yaml

To help starting/stopping vms and connecting to consoles, we install virtctl (which is aware of contexts):

VERSION="v0.14.0"
wget https://github.com/kubevirt/kubevirt/releases/download/$VERSION/virtctl-$VERSION-linux-amd64
mv virtctl-$VERSION-linux-amd64 /usr/bin/virtctl
chmod +x /usr/bin/virtctl

KubeVirt Deployment Verification

Verify that all KubeVirt pods are running in the clusters:

$ for c in cluster1 cluster2; do kubectl get pods -n kubevirt --context ${c} ; done
NAME                               READY     STATUS    RESTARTS   AGE
virt-api-578cff4f56-2dsml          1/1       Running   0          3m
virt-api-578cff4f56-8mk27          1/1       Running   0          3m
virt-controller-7d8c4fbc4c-pfwll   1/1       Running   0          3m
virt-controller-7d8c4fbc4c-xvlvr   1/1       Running   0          3m
virt-handler-plfg7                 1/1       Running   0          3m
virt-operator-67c86544f7-pnjjk     1/1       Running   0          5m
NAME                               READY     STATUS    RESTARTS   AGE
virt-api-578cff4f56-jjbmf          1/1       Running   0          3m
virt-api-578cff4f56-m6g2c          1/1       Running   0          3m
virt-controller-7d8c4fbc4c-tt9tz   1/1       Running   0          3m
virt-controller-7d8c4fbc4c-zf6hh   1/1       Running   0          3m
virt-handler-bldss                 1/1       Running   0          3m
virt-operator-67c86544f7-zz5jc     1/1       Running   0          5m

Now that KubeVirt is up and created its own custom resource types, we federate them:

kubefed2 enable virtualmachines
kubefed2 enable virtualmachineinstances
kubefed2 enable virtualmachineinstancepresets
kubefed2 enable virtualmachineinstancereplicasets
kubefed2 enable virtualmachineinstancemigrations

For demo purposes, we also federate persistent volume claims:

kubefed2 enable persistentvolumeclaim

Demo Workflow

We create a federated persistent volume claim, pointing to an existing pv created at both sites, against the same nfs server:

kubectl create -f federated_pvc.yaml

We then create a federated virtualmachine, with a placement so that it’s only created at cluster1

kubectl create -f federated_vm.yaml

We can check how its underlying pod only got created at one site:

$ for c in cluster1 cluster2; do kubectl get pods --context ${c} ; done
NAME                          READY     STATUS    RESTARTS   AGE
virt-launcher-testvm2-9dq48   2/2       Running   0          6m
No resources found.

Once the vm is up, we connect to it and format its secondary disk, put some data there

Playing with placement resource, we have it stopping at cluster1 and launch at cluster2.

kubectl patch federatedvirtualmachineplacements  testvm2 --type=merge -p '{"spec":{"clusterNames": ["cluster2"]}}'

We can then connect there and see how the data is still available!!!

Final Thoughts

Federating KubeVirt allows interesting use cases around kubevirt like disaster recovery scenarios.

More over, the pattern used to federate this product can be seen as a generic way to federate modern applications:

• federate operator
• federate the CRD deploying the app (either at both sites or selectively)
• federate the CRDS handled by the app