Hyper Converged Operator

HCO known as Hyper Converged Operator

Prerequisites:

This Blog assumes that the reader is aware of the concept of Operators and how it works in K8’s environment. Before proceeding further, feel free to take a look at this concept using CoreOS BlogPost

What it does?

The goal of the hyperconverged-cluster-operator (HCO) is to provide a single entrypoint for multiple operators - kubevirt, cdi, networking, etc… - where users can deploy and configure them in a single object. This operator is sometimes referred to as a “meta operator” or an “operator for operators”. Most importantly, this operator doesn’t replace or interfere with OLM. It only creates operator CRs, which is the user’s prerogative.

How does it work?

In this blog post, I’d like to focus on the first method(i.e by deploying a HCO using a CustomResourceDefinition method)which might seem like the most immediate benefit of this feature. Let’s get started!

Environment description

We can use HCO both on minikube and also on Openshift 4. We will be using Openshift 4 for HCO in this Blog.

Note: All the commands for installing HCO on minikube will remain the same as documented below, please follow the link Install_HCO_minikube install minikube by adjusting the memory to your requirement(atleast 4GiB of RAM is recommended).

Deploying HCO on Openshift 4 Cluster.

Openshift

Installation steps for Openshift 4 including video tutorial can be found here

Upon successful installation of OpenShift, we will have a cluster consisting of 3 masters and 3 workers which can be used for HCO integration

$oc version
Client Version: version.Info{Major:"4", Minor:"1+", GitVersion:"v4.1.0", GitCommit:"2793c3316", GitTreeState:"", BuildDate:"2019-04-23T07:46:06Z", GoVersion:"", Compiler:"", Platform:""}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.4+0ba401e", GitCommit:"0ba401e", GitTreeState:"clean", BuildDate:"2019-03-31T22:28:12Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

Check the nodes:

$oc get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-133-213.us-east-2.compute.internal Ready worker 12m v1.13.4+da48e8391
ip-10-0-138-120.us-east-2.compute.internal Ready master 18m v1.13.4+da48e8391
ip-10-0-146-51.us-east-2.compute.internal Ready master 18m v1.13.4+da48e8391
ip-10-0-150-215.us-east-2.compute.internal Ready worker 12m v1.13.4+da48e8391
ip-10-0-160-201.us-east-2.compute.internal Ready master 17m v1.13.4+da48e8391
ip-10-0-168-28.us-east-2.compute.internal Ready worker 12m v1.13.4+da48e8391

Clone the HCO repo:

git clone https://github.com/kubevirt/hyperconverged-cluster-operator.git

This gives all the necessary go packages and yaml manifests for the next steps.

Let’s create a NameSpace for the HCO deployment

oc create new-project kubevirt-hyperconverged

Now switch to the kubevirt-hyperconverged NameSpace

oc project kubevirt-hyperconverged

Now launch all the CRD’s

oc create -f deploy/converged/crds/hco.crd.yaml
oc create -f deploy/converged/crds/kubevirt.crd.yaml
oc create -f deploy/converged/crds/cdi.crd.yaml
oc create -f deploy/converged/crds/cna.crd.yaml

Let’s see the yaml file for HCO Custom Resource Definition

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: hyperconvergeds.hco.kubevirt.io
spec:
  additionalPrinterColumns:
  - JSONPath: .metadata.creationTimestamp
    name: Age
    type: date
  - JSONPath: .status.phase
    name: Phase
    type: string
  group: hco.kubevirt.io
  names:
    kind: HyperConverged
    plural: hyperconvergeds
    shortNames:
    - hco
    - hcos
    singular: hyperconverged
  scope: Namespaced
  subresources:
    status: {}
  version: v1alpha1
  versions:
  - name: v1alpha1
    served: true
    storage: true

Let’s create ClusterRoleBindings, ClusterRole, ServerAccounts and Deployments for the operator

$ oc create -f deploy/converged

And after verifying all the above resources we can now finally deploy our HCO custom resource

$ oc create -f deploy/converged/crds/hco.cr.yaml

We can take a look at the YAML definition of the CustomResource of HCO:

Let’s create ClusterRoleBindings, ClusterRole, ServerAccounts and Deployments for the operator

$ oc create -f deploy/converged

And after verifying all the above resources we can now finally deploy our HCO custom resource

$ oc create -f deploy/converged/crds/hco.cr.yaml

We can take a look at the YAML definition of the CustomResource of HCO:

---
apiVersion: hco.kubevirt.io/v1alpha1
kind: HyperConverged
metadata:
  name: hyperconverged-cluster

After successfully executing the above commands,we should be now be having a virt-controller pod, HCO pod, and a network-addon pod functional and can be viewed as below.

Let’s see the deployed pods:

$oc get pods
NAME                                               READY   STATUS    RESTARTS   AGE
cdi-apiserver-769fcc7bdf-rv8zt                     1/1     Running   0          5m2s
cdi-deployment-8b64c5585-g7zfk                     1/1     Running   0          5m2s
cdi-operator-c77447cc7-58ld2                       1/1     Running   0          11m
cdi-uploadproxy-8dcdcbff-rddl6                     1/1     Running   0          5m2s
cluster-network-addons-operator-85cd468ff5-xjgds   1/1     Running   0          11m
hyperconverged-cluster-operator-75dd9c96f9-pqvdk   1/1     Running   0          11m
virt-api-7f5bfb4c58-bkbhq                          1/1     Running   0          4m59s
virt-api-7f5bfb4c58-kkvwc                          1/1     Running   1          4m59s
virt-controller-6ccbfb7d5b-m7ljf                   1/1     Running   0          3m49s
virt-controller-6ccbfb7d5b-mbvlv                   1/1     Running   0          3m49s
virt-handler-hqz9d                                 1/1     Running   0          3m49s
virt-operator-667b6c845d-jfnsr                     1/1     Running   0          11m

Also the below deployments:

$oc get deployments
NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
cdi-apiserver                     1/1     1            1           10m
cdi-deployment                    1/1     1            1           10m
cdi-operator                      1/1     1            1           16m
cdi-uploadproxy                   1/1     1            1           10m
cluster-network-addons-operator   1/1     1            1           16m
hyperconverged-cluster-operator   1/1     1            1           16m
virt-api                          2/2     2            2           9m58s
virt-controller                   2/2     2            2           8m49s
virt-operator                     1/1     1            1           16m

Note: Here, Once we applied the Custom Resource the operator took care of deploying the actual KubeVirt pods (virt-api, virt-controller and virt-handler), CDI pods(cdi-upload-proxy, cdi-apiserver, cdi-deployment, cdi-operator) and Network add-on pods ( cluster-network-addons-operator).We will need to wait until all of the resources are up and running. This can be done using the command above or by using the command above with the -wflag.

After the HCO is up and running on the cluster, we should be able to see the info of CRD’s

$oc get crds | grep kubevirt
cdiconfigs.cdi.kubevirt.io                                       2019-05-07T20:22:17Z
cdis.cdi.kubevirt.io                                             2019-05-07T20:20:58Z
datavolumes.cdi.kubevirt.io                                      2019-05-07T20:22:17Z
hyperconvergeds.hco.kubevirt.io                                  2019-05-07T20:20:58Z
kubevirtcommontemplatesbundles.kubevirt.io                       2019-05-07T20:20:58Z
kubevirtnodelabellerbundles.kubevirt.io                          2019-05-07T20:20:58Z
kubevirts.kubevirt.io                                            2019-05-07T20:20:58Z
kubevirttemplatevalidators.kubevirt.io                           2019-05-07T20:20:58Z
kwebuis.kubevirt.io                                              2019-05-07T20:20:58Z
networkaddonsconfigs.networkaddonsoperator.network.kubevirt.io   2019-05-07T20:20:58Z
nodemaintenances.kubevirt.io                                     2019-05-07T20:20:58Z
virtualmachineinstancemigrations.kubevirt.io                     2019-05-07T20:23:02Z
virtualmachineinstancepresets.kubevirt.io                        2019-05-07T20:23:01Z
virtualmachineinstancereplicasets.kubevirt.io                    2019-05-07T20:23:02Z
virtualmachineinstances.kubevirt.io                              2019-05-07T20:23:01Z
virtualmachines.kubevirt.io                                      2019-05-07T20:23:02Z

Note: In Openshift we can use both kubectl and oc interchangeably to interact with the cluster objects once HCO is up and running.

You can also read more about CDI, CNA, ssp-operator, web-ui and KubeVirt:

HCO using the OLM method

Note: The complete architecture of OLM and its components that connect together can be understood using the link OLM_architecture

Replace with your Docker organization as official operator-registry images for HCO will not be provided.

Next, build and publish the converged HCO operator-registry image.

cd deploy/converged
export HCO_DOCKER_ORG=<docker_org>
docker build --no-cache -t docker.io/$HCO_DOCKER_ORG/hco-registry:example -f Dockerfile .
docker push docker.io/$HCO_DOCKER_ORG/hco-registry:example

As an example deployment, Let’s take the value of operator-registry image as

docker.io/rthallisey/hyperconverged-cluster-operator:latest

Now, Let’s create the kubevirt-hyperconverged NS as below

oc create ns kubevirt-hyperconverged

Create the OperatorGroup

cat <<EOF | oc create -f -
apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
  name: hco-operatorgroup
  namespace: kubevirt-hyperconverged
EOF

Create a Catalog Source backed by a grpc registry

cat <<EOF | oc create -f -
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: hco-catalogsource
  namespace: openshift-operator-lifecycle-manager
  imagePullPolicy: Always
spec:
  sourceType: grpc
  image: docker.io/rthallisey/hco-registry:v0.1-8
  displayName: KubeVirt HyperConverged
  publisher: Red Hat
EOF

Please wait until the hco-catalogsource pod comes up

Next is to create a subscription, we can create a subscription from the Openshift4 web interface as shown below:

Subscription

Once subscribed, we can create a kubevirt Hyperconverged Operator from UI:

Creating-Subscription

Install the HCO Operator:

Installed-HCO

Please wait until the virt-operator, cdi-operator and cluster-network-addons-operator comes up.

After they are up, its now time to launch the HCO-Custom Resource itself:

HCO-CR

Once the HCO Operator is deployed in the kubevirt-hyperconverged NS, we can see all the pods are up and running:

HCO-Managed-Operators

We can verify the same from the CLI:

oc get pods -n kubevirt-hyperconverged
NAME                                               READY   STATUS    RESTARTS   AGE
cdi-apiserver-769fcc7bdf-b5v8n                     1/1     Running   0          4m5s
cdi-deployment-8b64c5585-qs527                     1/1     Running   0          4m4s
cdi-operator-77b8847b96-5kmb2                      1/1     Running   0          4m55s
cdi-uploadproxy-8dcdcbff-xgnxf                     1/1     Running   0          4m5s
cluster-network-addons-operator-584dff99b8-c5kz5   1/1     Running   0          4m55s
hco-operator-59b559bd44-lgdnm                      1/1     Running   0          4m55s
kubevirt-ssp-operator-67b78446f7-l7rfv             1/1     Running   0          4m55s
kubevirt-web-ui-operator-9df6b67d9-mzf6s           1/1     Running   0          4m55s
node-maintenance-operator-6b464dc85-v6vmw          1/1     Running   0          4m55s
virt-api-7b56d7dd89-8s78r                          1/1     Running   0          2m59s
virt-api-7b56d7dd89-h75t8                          1/1     Running   1          2m59s
virt-controller-77c6d6d779-9qpp4                   1/1     Running   0          2m32s
virt-controller-77c6d6d779-vbbxg                   1/1     Running   0          2m32s
virt-handler-4bfb9                                 1/1     Running   0          2m32s
virt-handler-ns97x                                 1/1     Running   0          2m32s
virt-handler-q7wbh                                 1/1     Running   0          2m32s
virt-operator-87d7c98b-mh8pg                       1/1     Running   0          4m55s
virt-operator-87d7c98b-p6mbd                       1/1     Running   0          4m55s

We can see how OLM operator manages the HCO pods from the openshift-operator-lifecycle-manager NS:

HCO-Managed-Operators

The above method demonstrates the integration of HCO operator in Openshift4.

So, after HCO is up and running we need to test it by deploying a small instance of a VM.To deploy an instance follow the instructions here minikube_quickstart:

Conclusion

What to expect next?

HCO achieved its goal which was to provide a single entrypoint for multiple operators - kubevirt, cdi, networking, etc.where users can deploy and configure them in a single object as seen above.

Now, we can also launch the HCO through OLM.

Note: Until we publish (and consume) the HCO and component operators through operatorhub.io, this is a means to demonstrate the HCO workflow without OLM

Once we publish operators through Marketplace at , it will be available here