A large number of virt-launcher Pods remain in the Failed phase.
Condition:
sum(kube_pod_status_phase{phase="Failed", pod=~"virt-launcher-.*"}) >= 200
holds for 10 minutes.
Virt-launcher Pods host virtual machine (VM) workloads and mass failures
can indicate migration loops, image/network/storage issues, or
control-plane regressions.
VMs and the cluster control plane might be affected:
sum(kube_pod_status_phase{phase="Failed", pod=~"virt-launcher-.*"})
count by (namespace) (
max by (namespace, pod) (kube_pod_status_phase{phase="Failed", pod=~"virt-launcher-.*"}) == 1
)
```promql count by (node) ( (max by (namespace, pod) (kube_pod_status_phase{phase=”Failed”, pod=~”virt-launcher-.*”}) == 1)
max by aggregations collapse duplicate series for each Pod, including
kube_pod_info series that differ only by pod_ip. Matching on both
namespace and pod keeps Pods with the same name in different namespaces
separate. topk(5, count by (reason) (kube_pod_container_status_last_terminated_reason{pod=~"virt-launcher-.*"} == 1))
# List a few failed virt-launcher pods cluster-wide
$ kubectl get pods -A -l kubevirt.io=virt-launcher --field-selector=status.phase=Failed --no-headers | head -n 20
# Inspect events for a representative pod (image/CNI/storage/useful errors)
$ kubectl -n <namespace> describe pod <virt-launcher-pod> | sed -n '/Events/,$p'
$ kubectl get vmim -A
NAMESPACE="$(kubectl get kubevirt -A -o jsonpath='{.items[0].metadata.namespace}')"
$ kubectl -n "$NAMESPACE" logs -l kubevirt.io=virt-controller --tail=200
$ kubectl -n "$NAMESPACE" logs -l kubevirt.io=virt-handler --tail=200
ImagePullBackOff eventsReduce the blast radius:
$ kubectl delete vmim -A
Clean up failed Pods (relieves API/etcd and monitoring):
This deletes Pods across every namespace, so preview the Pods that
match the selectors first. The listing includes the namespace of
each Pod, so you can confirm that only Failed virt-launcher Pods
match, and not running VM workloads:
$ kubectl get pods -A -l kubevirt.io=virt-launcher \
--field-selector=status.phase=Failed
Only after verifying that listing, delete the Pods with the same selectors:
$ kubectl delete pods -A -l kubevirt.io=virt-launcher \
--field-selector=status.phase=Failed
Resolve root cause:
Validate resolution (alert clears):
sum(kube_pod_status_phase{phase="Failed", pod=~"virt-launcher-.*"})
Ensure the failed count drops and stays below threshold and that new virt-launcher Pods start successfully and VMIs are healthy.
If you cannot resolve the issue, see the following resources: