Why is KEDA API metrics server failing when Istio is installed?
While setting up KEDA, you get an error: (v1beta1.external.metrics.k8s.io) status FailedDiscoveryCheck
and you have Istio installed as service mesh in your cluster.
This can lead to side effects like not being able to delete namespaces in your cluster. You will see an error like:
NamespaceDeletionDiscoveryFailure - Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
Check the setup
In the following we assume that KEDA is installed in the namespace keda
.
Check the KEDA API service status
Find the api service name for the service keda/keda-metrics-apiserver
:
kubectl get apiservice --all-namespaces
Check for the status of the api service found in previous step:
kubectl get apiservice <apiservicename> -o yaml
Example:
kubectl get apiservice v1beta1.external.metrics.k8s.io -o yaml
If the status is False
, then there seems to be an issue with the KEDA installation.
Check Istio installation
Check if Istio is installed in your cluster:
kubectl get svc -n istio-system
If Istio is installed you get a result like:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-ingressgateway LoadBalancer 100.65.18.245 34.159.50.243 15021:31585/TCP,80:31669/TCP,443:30464/TCP 3d
istiod ClusterIP 100.65.146.141 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP 3d
Check KEDA namespace labels
Check the KEDA namespace labels:
kubectl describe ns keda
If Istio injection is enabled there is no label stating istio-injection=disabled
.
In this setup the sidecar injection of Istio prevents the api service of KEDA to work properly.
Solution
To prevent the side-car injection of Istio we must label the namespace accordingly. This can be achieved via setting the label istio-injection=disabled
to the namespace:
kubectl label namespace keda istio-injection=disabled
Check that the label is set via kubectl describe ns keda
Install KEDA into the namespace keda
and re-check the status of the api service which should now have the status True
.