Health Checking of Istio Services

This task shows how to use Kubernetes liveness and readiness probes for health checking of Istio services.

There are three options for liveness and readiness probes in Kubernetes:

  1. Command
  2. HTTP request
  3. TCP request

This task provides examples for the first two options with Istio mutual TLS enabled and disabled, respectively.

Before you begin

Liveness and readiness probes with command option

In this section, you configure health checking when mutual TLS is disabled, then when mutual TLS is enabled.

Mutual TLS disabled

Run this command to deploy liveness in the default namespace:

Zip
$ kubectl apply -f <(istioctl kube-inject -f @samples/health-check/liveness-command.yaml@)

Wait for a minute and check the pod status:

$ kubectl get pod
NAME                             READY     STATUS    RESTARTS   AGE
liveness-6857c8775f-zdv9r        2/2       Running   0           1m

The number ‘0’ in the ‘RESTARTS’ column means liveness probes worked fine. Readiness probes work in the same way and you can modify liveness-command.yaml accordingly to try it yourself.

Mutual TLS enabled

To enable mutual TLS for services in the default namespace, you must configure an authentication policy and a destination rule. Follow these steps to complete the configuration:

  1. To configure the authentication policy, run:

    $ kubectl apply -f - <<EOF
    apiVersion: "authentication.istio.io/v1alpha1"
    kind: "Policy"
    metadata:
      name: "default"
      namespace: "default"
    spec:
      peers:
      - mtls: {}
    EOF
    
  2. To configure the destination rule, run:

    $ kubectl apply -f - <<EOF
    apiVersion: "networking.istio.io/v1alpha3"
    kind: "DestinationRule"
    metadata:
      name: "default"
      namespace: "default"
    spec:
      host: "*.default.svc.cluster.local"
      trafficPolicy:
        tls:
          mode: ISTIO_MUTUAL
    EOF
    

Run this command to re-deploy the service:

ZipZip
$ kubectl delete -f <(istioctl kube-inject -f @samples/health-check/liveness-command.yaml@)
$ kubectl apply -f <(istioctl kube-inject -f @samples/health-check/liveness-command.yaml@)

Repeat the check status command to verify that the liveness probes work:

$ kubectl get pod
NAME                             READY     STATUS    RESTARTS   AGE
liveness-6857c8775f-zdv9r        2/2       Running   0           4m

Cleanup

Remove the mutual TLS policy and corresponding destination rule added in the steps above:

  1. To remove the mutual TLS policy, run:

    $ kubectl delete policies default
    
  2. To remove the corresponding destination rule, run:

    $ kubectl delete destinationrules default
    

Liveness and readiness probes with HTTP request option

This section shows how to configure health checking with the HTTP request option.

Mutual TLS is disabled

Run this command to deploy liveness-http in the default namespace:

Zip
$ kubectl apply -f <(istioctl kube-inject -f @samples/health-check/liveness-http.yaml@)

Wait for a minute and check the pod status to make sure the liveness probes work with ‘0’ in the ‘RESTARTS’ column.

$ kubectl get pod
NAME                             READY     STATUS    RESTARTS   AGE
liveness-http-975595bb6-5b2z7c   2/2       Running   0           1m

Mutual TLS is enabled

When mutual TLS is enabled, we have two options to support HTTP probes: probe rewrites and separate ports.

Probe rewrite

This approach rewrites the application PodSpec liveness probe, such that the probe request will be sent to Pilot agent. Pilot agent then redirects the request to application, and strips the response body only returning the response code.

To use this approach, you need to install Istio with Helm option sidecarInjectorWebhook.rewriteAppHTTPProbe=true. Note this is a global flag. Turning it on means all Istio app deployment will be affected. Please be aware of the risk.

Zip
$ helm template install/kubernetes/helm/istio --name istio --namespace istio-system \
    --set global.mtls.enabled=true --set sidecarInjectorWebhook.rewriteAppHTTPProbe=true \
    -f @install/kubernetes/helm/istio/values.yaml@ > $HOME/istio.yaml
$ kubectl apply -f $HOME/istio.yaml

Re-deploy the liveness health check app.

The above Helm configuration makes it so sidecar injection automatically rewrites the Kubernetes pod YAML, such that health checks can work under mutual TLS. No need to update your app or Pod YAML by yourself.

ZipZip
$ kubectl delete -f <(istioctl kube-inject -f @samples/health-check/liveness-command.yaml@)
$ kubectl apply -f <(istioctl kube-inject -f @samples/health-check/liveness-command.yaml@)
$ kubectl get pod
NAME                             READY     STATUS    RESTARTS   AGE
liveness-http-975595bb6-5b2z7c   2/2       Running   0           1m

This features is not currently turned on by default. We’d like to hear your feedback on whether we should change this to default behavior for Istio installation.

Separate port

Again, enable mutual TLS for services in the default namespace by adding namespace-wide authentication policy and a destination rule:

  1. To configure the authentication policy, run:

    $ kubectl apply -f - <<EOF
    apiVersion: "authentication.istio.io/v1alpha1"
    kind: "Policy"
    metadata:
      name: "default"
      namespace: "default"
    spec:
      peers:
      - mtls: {}
    EOF
    
  2. To configure the destination rule, run:

    $ kubectl apply -f - <<EOF
    apiVersion: "networking.istio.io/v1alpha3"
    kind: "DestinationRule"
    metadata:
      name: "default"
      namespace: "default"
    spec:
      host: "*.default.svc.cluster.local"
      trafficPolicy:
        tls:
          mode: ISTIO_MUTUAL
    EOF
    

Run these commands to re-deploy the service:

ZipZip
$ kubectl delete -f <(istioctl kube-inject -f @samples/health-check/liveness-http.yaml@)
$ kubectl apply -f <(istioctl kube-inject -f @samples/health-check/liveness-http.yaml@)

Wait for a minute and check the pod status to make sure the liveness probes work with ‘0’ in the ‘RESTARTS’ column.

$ kubectl get pod
NAME                             READY     STATUS    RESTARTS   AGE
liveness-http-67d5db65f5-765bb   2/2       Running   0          1m

Note that the image in liveness-http exposes two ports: 8001 and 8002 (source code). In this deployment, port 8001 serves the regular traffic while port 8002 is used for liveness probes. Because the Istio proxy only intercepts ports that are explicitly declared in the containerPort field, traffic to 8002 port bypasses the Istio proxy regardless of whether Istio mutual TLS is enabled. However, if you use port 8001 for both regular traffic and liveness probes, health check will fail when mutual TLS is enabled because the HTTP request is sent from Kubelet, which does not send client certificate to the liveness-http service.