monitoring

VirtLauncherPodsStuckFailed

Meaning

A large number of virt-launcher Pods remain in the Failed phase.

Condition: sum(kube_pod_status_phase{phase="Failed", pod=~"virt-launcher-.*"}) >= 200 holds for 10 minutes. Virt-launcher Pods host virtual machine (VM) workloads and mass failures can indicate migration loops, image/network/storage issues, or control-plane regressions.

Impact

VMs and the cluster control plane might be affected:

Diagnosis

  1. Confirm scope and distribution:
     sum(kube_pod_status_phase{phase="Failed", pod=~"virt-launcher-.*"})
    
     count by (namespace) (
       max by (namespace, pod) (kube_pod_status_phase{phase="Failed", pod=~"virt-launcher-.*"}) == 1
     )
    

    ```promql count by (node) ( (max by (namespace, pod) (kube_pod_status_phase{phase=”Failed”, pod=~”virt-launcher-.*”}) == 1)

    • on(namespace, pod) group_left(node) max by (namespace, pod, node) (kube_pod_info{pod=~”virt-launcher-.*”, node!=””}) ) ``` The max by aggregations collapse duplicate series for each Pod, including kube_pod_info series that differ only by pod_ip. Matching on both namespace and pod keeps Pods with the same name in different namespaces separate.
     topk(5, count by (reason) (kube_pod_container_status_last_terminated_reason{pod=~"virt-launcher-.*"} == 1))
    
  2. Sample failed Pods and events:
     # List a few failed virt-launcher pods cluster-wide
     $ kubectl get pods -A -l kubevirt.io=virt-launcher --field-selector=status.phase=Failed --no-headers | head -n 20
    
     # Inspect events for a representative pod (image/CNI/storage/useful errors)
     $ kubectl -n <namespace> describe pod <virt-launcher-pod> | sed -n '/Events/,$p'
    
  3. Check for migration storms:
     $ kubectl get vmim -A
    
  4. In the control plane and component logs, look for spikes/errors:
     NAMESPACE="$(kubectl get kubevirt -A -o jsonpath='{.items[0].metadata.namespace}')"
    
     $ kubectl -n "$NAMESPACE" logs -l kubevirt.io=virt-controller --tail=200
    
     $ kubectl -n "$NAMESPACE" logs -l kubevirt.io=virt-handler --tail=200
    
  5. Check the infrastructure for common causes:

Mitigation

  1. Reduce the blast radius:

    • Migration loop: cancel in-flight migrations (scope as needed).
      $ kubectl delete vmim -A
      
    • Coordinate with noisy tenants; pause offending workloads if necessary.
  2. Clean up failed Pods (relieves API/etcd and monitoring):

    This deletes Pods across every namespace, so preview the Pods that match the selectors first. The listing includes the namespace of each Pod, so you can confirm that only Failed virt-launcher Pods match, and not running VM workloads:

     $ kubectl get pods -A -l kubevirt.io=virt-launcher \
         --field-selector=status.phase=Failed
    

    Only after verifying that listing, delete the Pods with the same selectors:

     $ kubectl delete pods -A -l kubevirt.io=virt-launcher \
         --field-selector=status.phase=Failed
    
  3. Resolve root cause:

    • Image issues: fix registry access, credentials, or tags; re-run affected workloads.
    • Network/CNI: fix CNI/data-plane errors; confirm new Pods start cleanly.
    • Storage: resolve attach/mount failures; verify PVC/VolumeSnapshot health.
    • KubeVirt regression: roll forward/back to a known-good version and re-try.
  4. Validate resolution (alert clears):

     sum(kube_pod_status_phase{phase="Failed", pod=~"virt-launcher-.*"})
    

Ensure the failed count drops and stays below threshold and that new virt-launcher Pods start successfully and VMIs are healthy.

If you cannot resolve the issue, see the following resources: