How can I enable/disable mutual TLS after I installed Istio?
How can I check whether mutual TLS is enabled for a service?
istioctl command provides an option for this purpose. You can do:
$ istioctl authn tls-check $CLIENT_POD httpbin.default.svc.cluster.local HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE httpbin.default.svc.cluster.local:8000 OK STRICT ISTIO_MUTUAL /default istio-system/default
$CLIENT_POD is the ID of one of the client service’s pods.
Refer to Verify mutual TLS configuration for more information.
Can I enable mutual TLS for some services while leaving it disabled for other services in the same cluster?
Authentication policy can be mesh-wide (which affects all services in the mesh), namespace-wide (all services in the same namespace) or service specific. You can have policy or policies to setup mutual TLS for services in a cluster in any way as you want.
If mutual TLS is globally enabled, can non-Istio services access Istio services?
Non-Istio services cannot communicate to Istio services unless they can present a valid certificate, which is less likely to happen. This is the expected behavior for mutual TLS. However, you can override the global flag for specific namespaces or services. See task for more details.
How can services that use Istio access non-Istio services?
When mutual TLS is globally enabled, the global destination rule matches all services in the cluster, whether or not these services have an Istio sidecar. This includes the Kubernetes API server, as well as any non-Istio services in the cluster. To communicate with such services from services that have an Istio sidecar, you need to set a destination rule to exempt the service. For example:
$ kubectl apply -f - <<EOF apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: "api-server" spec: host: "kubernetes.default.svc.cluster.local" trafficPolicy: tls: mode: DISABLE EOF
Similarly, you can add destination rules for other non-Istio services. For more examples, see task.
How can I use Kubernetes liveness and readiness for pod health checks when mutual TLS is enabled?
If mutual TLS is enabled, HTTP and TCP health checks from the kubelet will not work without modification, since the kubelet does not have Istio-issued certificates.
As of Istio 1.1, we have several options to solve this issue.
Using probe rewrite to redirect liveness and readiness requests to the workload directly. Please refer to Probe Rewrite for more information.
Using a separate port for health checks and enabling mutual TLS only on the regular service port. Please refer to Health Checking of Istio Services for more information.
PERMISSIVEmode for Istio services so they can accept both HTTP and mutual TLS traffic. Please keep in mind that mutual TLS is not enforced since others can communicate with the service with HTTP traffic.
Using a liveness command for health checks, e.g., one can install
curlin the service pod and
curlitself within the pod.
An example of a readiness probe:
livenessProbe: exec: command: - curl - -f - http://localhost:8080/healthz # Replace port and URI by your actual health check initialDelaySeconds: 10 periodSeconds: 5
How to configure the lifetime for Istio certificates?
For the workloads running in Kubernetes, the lifetime of their Istio certificates is controlled by the
workload-cert-ttl flag on Citadel. The default value is 90 days. This value should be no greater than
max-workload-cert-ttl of Citadel.
Citadel uses a flag
max-workload-cert-ttl to control the maximum lifetime for Istio certificates issued to
workloads. The default value is 90 days. If
workload-cert-ttl on Citadel or node agent is greater than
max-workload-cert-ttl, Citadel will fail issuing the certificate.
You can modify a generated manifest file to customize the Citadel configuration. The following modification specifies that the Istio certificates for workloads running in Kubernetes has 1 hours lifetime. Besides that, the maximum allowed Istio certificate lifetime is 48 hours.
... kind: Deployment ... metadata: name: istio-citadel namespace: istio-system spec: ... template: ... spec: ... containers: - name: citadel ... args: - --workload-cert-ttl=1h # Lifetime of certificates issued to workloads in Kubernetes. - --max-workload-cert-ttl=48h # Maximum lifetime of certificates issued to workloads by Citadel.
For the workloads running on VMs and bare metal hosts, the lifetime of their Istio certificates is specified by the
workload-cert-ttl flag on each node agent. The default value is also 90 days. This value should be no greater than
max-workload-cert-ttl of Citadel.
To customize this configuration, the argument for the node agent service should be modified.
After setting up the machines for Istio
mesh expansion, modify the file
/lib/systemd/system/istio-auth-node-agent.service on the VMs or bare metal hosts:
... [Service] ExecStart=/usr/local/bin/node_agent --workload-cert-ttl=24h # Specify certificate lifetime for workloads on this machine. Restart=always StartLimitInterval=0 RestartSec=10 ...
The above configuration specifies that the Istio certificates for workloads running on this VM or bare metal host will have 24 hours lifetime.
After configuring the service, restart the node agent by running
MySQL Connectivity Troubleshooting
You may find MySQL can’t connect after installing Istio. This is because of
which is enabled in the
demo configuration profile,
does not work with MySQL.
You may see error messages such as
ERROR 2013 (HY000): Lost connection to MySQL server at
'reading initial communication packet', system error: 0.
There have two options to solve the problem.
Disable Mutual TLS.
Choose this option if you don’t want Istio mutual TLS. You achieve this by disabling mutual TLS on the MySQL service explicitly.
$ kubectl apply -f - <<EOF apiVersion: "authentication.istio.io/v1alpha1" kind: "Policy" metadata: name: mysql-nomtls-authn spec: targets: - name: YOUR-MYSQL-SERVICE # The name of *your* K8s Service EOF
Enable mutual TLS in STRICT mode.
If you want mutual TLS protection for MySQL, enable mutual TLS using a destination rule and an authentication policy.
$ kubectl apply -f - <<EOF apiVersion: "authentication.istio.io/v1alpha1" kind: "Policy" metadata: name: mysql-mtls-authn spec: targets: - name: YOUR-MYSQL-SERVICE # The name of *your* K8s Service peers: - mtls: mode: STRICT --- apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: mysql-mtls-dr spec: host: YOUR-MYSQL-SERVICE # The name of *your* K8s Service trafficPolicy: tls: mode: ISTIO_MUTUAL EOF
Does Istio support authorization?
Yes. Istio provides authorization features for both HTTP and plain TCP services in the mesh. Learn more.
Does Istio authentication use Kubernetes secrets?
Yes. The key and certificate distribution in Istio Authentication is based on Kubernetes secrets.
Secrets have known security risks. The Kubernetes team is working on several features to improve Kubernetes secret security, from secret encryption to node-level access control. And as of version 1.6, Kubernetes introduces RBAC authorization, which can provide fine-grained secrets management.
Is the secret encrypted for workload key and cert?
By default, they are base64 encoded but not encrypted. However, the secret encryption feature is supported in Kubernetes and you can do it by following the instruction.
Notice that this feature is not enabled yet in Google Container Engine (GKE). While the data may not be encrypted inside the etcd running on the master node, the contents of the master node itself are encrypted, see here for more info.
How to configure Istio Ingress to only accept TLS traffic?
By following the instructions in the Secure Ingress Traffic task, Istio Ingress can be secured to only accept TLS traffic.
Can I install Istio sidecar for HTTPS services?
Yes, you can. It works both with mutual TLS enabled and disabled. Refer to how mutual TLS works with HTTPS services for more information.