Load-balancer for virtual machines on bare metal Kubernetes clusters

Introduction

Over the last year, Kubevirt and MetalLB have shown to be powerful duo in order to support fault-tolerant access to an application on virtual machines through an external IP address. As a Cluster administrator using an on-prem cluster without a network load-balancer, now it’s possible to use MetalLB operator to provide load-balancer capabilities (with Services of type LoadBalancer) to virtual machines.

MetalLB

MetalLB allows you to create Kubernetes services of type LoadBalancer, and provides network load-balancer implementation in on-prem clusters that don’t run on a cloud provider. MetalLB is responsible for assigning/unassigning an external IP Address to your service, using IPs from pre-configured pools. In order for the external IPs to be announced externally, MetalLB works in 2 modes, Layer 2 and BGP:

Layer 2 mode (ARP/NDP):

This mode - which actually does not implement real load-balancing behavior - provides a failover mechanism where a single node owns the LoadBalancer service, until it fails, triggering another node to be chosen as the service owner. This configuration mode makes the IPs reachable from the local network.
In this method, the MetalLB speaker pod announces the IPs in ARP (for IPv4) and NDP (for IPv6) protocols over the host network. From a network perspective, the node owning the service appears to have multiple IP addresses assigned to a network interface. After traffic is routed to the node, the service proxy sends the traffic to the application pods.
BGP mode:

This mode provides real load-balancing behavior, by establishing BGP peering sessions with the network routers - which advertise the external IPs of the LoadBalancer service, distributing the load over the nodes.

To read more on MetalLB concepts, implementation and limitations, please read its documentation.

Demo: Virtual machine with external IP and MetalLB load-balancer

With the following recipe we will end up with a nginx server running on a virtual machine, accessible outside the cluster using MetalLB load-balancer with Layer 2 mode.

Demo environment setup

We are going to use kind provider as an ephemeral Kubernetes cluster.

Prerequirements:

First install kind on your machine following its installation guide.
To use kind, you will also need to install docker.

External IPs on macOS and Windows

This demo runs Docker on Linux, which allows sending traffic directly to the load-balancer’s external IP if the IP space is within the docker IP space. On macOS and Windows however, docker does not expose the docker network to the host, rendering the external IP unreachable from other kind nodes. In order to workaround this, one could expose pods and services using extra port mappings as shown in the extra port mappings section of kind’s Configuration Guide.

Deploying cluster

To start a kind cluster:

kind create cluster

In order to interact with the specific cluster created:

kubectl cluster-info --context kind-kind

Installing components

Installing MetalLB on the cluster

There are many ways to install MetalLB. For the sake of this example, we will install MetalLB via manifests. To do this, follow this guide. Confirm successful installation by waiting for MetalLB pods to have a status of Running:

kubectl get pods -n metallb-system --watch

Installing Kubevirt on the cluster

Following Kubevirt user guide to install released version v0.51.0

export RELEASE=v0.51.0
kubectl apply -f "https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-operator.yaml"
kubectl apply -f "https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-cr.yaml"
kubectl -n kubevirt wait kv kubevirt --timeout=360s --for condition=Available

Now we have a Kubernetes cluster with all the pieces to start the Demo.

Network resources configuration

Setting Address Pool to be used by the LoadBalancer

In order to complete the Layer 2 mode configuration, we need to set a range of IP addresses for the LoadBalancer to use. On Linux we can use the docker kind network (macOS and Windows users see External IPs Prerequirement), so by using this command:

docker network inspect -f '' kind

You should get the subclass you can set the IP range from. The output should contain a cidr such as 172.18.0.0/16. Using this result we will create the following Layer 2 address pool with 172.18.1.1-172.18.1.16 range:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: addresspool-sample1
      protocol: layer2
      addresses:
      - 172.18.1.1-172.18.1.16
EOF

Network utilization

Spin up a Virtual Machine running Nginx

Now it’s time to start-up a virtual machine running nginx using the following yaml. The virtual machine has a metallb-service=nginx we created to use when creating the service.

cat <<EOF | kubectl apply -f -
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: fedora-nginx
  namespace: default
  labels:
    metallb-service: nginx
spec:
  running: true
  template:
    metadata:
      labels:
        metallb-service: nginx
    spec:
      domain:
        devices:
          disks:
            - disk:
                bus: virtio
              name: containerdisk
            - disk:
                bus: virtio
              name: cloudinitdisk
          interfaces:
            - masquerade: {}
              name: default
        resources:
          requests:
            memory: 1024M
      networks:
        - name: default
          pod: {}
      terminationGracePeriodSeconds: 0
      volumes:
        - containerDisk:
            image: kubevirt/fedora-cloud-container-disk-demo
          name: containerdisk
        - cloudInitNoCloud:
            userData: |-
              #cloud-config
              password: fedora
              chpasswd: { expire: False }
              packages:
                - nginx
              runcmd:
                - [ "systemctl", "enable", "--now", "nginx" ]
          name: cloudinitdisk
EOF

Expose the virtual machine with a typed `LoadBalancer` service

When creating the LoadBalancer typed service, we need to remember annotating the address-pool we want to use addresspool-sample1 and also add the selector metallb-service: nginx:

cat <<EOF | kubectl apply -f -
kind: Service
apiVersion: v1
metadata:
  name: metallb-nginx-svc
  namespace: default
  annotations:
    metallb.universe.tf/address-pool: addresspool-sample1
spec:
  externalTrafficPolicy: Local
  ipFamilies:
    - IPv4
  ports:
    - name: tcp-5678
      protocol: TCP
      port: 5678
      targetPort: 80
  type: LoadBalancer
  selector:
    metallb-service: nginx
EOF

Notice that the service got assigned with an external IP from the range assigned by the address pool:

kubectl get service -n default metallb-nginx-svc

Example output:

NAME                TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
metallb-nginx-svc   LoadBalancer   10.96.254.136   172.18.1.1    5678:32438/TCP   53s

Access the virtual machine from outside the cluster

Finally, we can check that the nginx server is accessible from outside the cluster:

curl -s -o /dev/null 172.18.1.1:5678 && echo "URL exists"