StatefulSets Made Easier With Istio 1.10

Learn how to easily deploy StatefulSets with Istio 1.10.

May 19, 2021 | By Lin Sun - Solo.io, Christian Posta - Solo.io, John Howard - Google, Zhonghu Xu - Huawei

Kubernetes StatefulSets are commonly used to manage stateful applications. In addition to managing the deployment and scaling of a set of Pods, StatefulSets provide guarantees about the ordering and uniqueness of those Pods. Common applications used with StatefulSets include ZooKeeper, Cassandra, Elasticsearch, Redis and NiFi.

The Istio community has been making gradual progress towards zero-configuration support for StatefulSets; from automatic mTLS, to eliminating the need to create DestinationRule or ServiceEntry resources, to the most recent pod networking changes in Istio 1.10.

What is unique about using a StatefulSet with a service mesh? The StatefulSet pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling. The kind of apps that run in a StatefulSet are often those that need to communicate among their pods, and, as they come from a world of hard-coded IP addresses, may listen on the pod IP only, instead of 0.0.0.0.

ZooKeeper, for example, is configured by default to not listen on all IPs for quorum communication:

quorumListenOnAllIPs=false

Over the last few releases, the Istio community has reported many issues around support for applications running in StatefulSets.

StatefulSets in action, prior to Istio 1.10

In a GKE cluster running Kubernetes 1.19, we have Istio 1.9.5 installed. We enabled automatic sidecar injection in the default namespace, then we installed ZooKeeper using the Helm charts provided by Bitnami, along with the Istio sleep pod for interactive debugging:

$ helm repo add bitnami https://charts.bitnami.com/bitnami $ helm install my-release bitnami/zookeeper --set replicaCount=3 $ kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/sleep/sleep.yaml

After a few minutes, all pods come up nicely with sidecar proxies:

$ kubectl get pods,svc
NAME READY STATUS RESTARTS AGE my-release-zookeeper-0 2/2 Running 0 3h4m my-release-zookeeper-1 2/2 Running 0 3h4m my-release-zookeeper-2 2/2 Running 0 3h5m pod/sleep-8f795f47d-qkgh4 2/2 Running 0 3h8m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE my-release-zookeeper ClusterIP 10.100.1.113 <none> 2181/TCP,2888/TCP,3888/TCP 3h my-release-zookeeper-headless ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 3h service/sleep ClusterIP 10.100.9.26 <none> 80/TCP 3h

Are our ZooKeeper services working and is the status Running? Let’s find out! ZooKeeper listens on 3 ports:

By default, the ZooKeeper installation configures port 2181 to listen on 0.0.0.0 but ports 2888 and 3888 only listen on the pod IP. Let’s check out the network status on each of these ports from one of the ZooKeeper pods:

$ kubectl exec my-release-zookeeper-1 -c istio-proxy -- netstat -na | grep -E '(2181|2888|3888)'
tcp 0 0 0.0.0.0:2181 0.0.0.0:* LISTEN tcp 0 0 10.96.7.7:3888 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:2181 127.0.0.1:37412 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:37486 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:37456 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:37498 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:37384 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:37514 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:37402 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:37434 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:37526 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:37374 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:37442 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:37464 TIME_WAIT

There is nothing ESTABLISHED on port 2888 or 3888. Next, let us get the ZooKeeper server status:

$ kubectl exec my-release-zookeeper-1 -c zookeeper -- /opt/bitnami/zookeeper/bin/zkServer.sh status
/opt/bitnami/java/bin/java ZooKeeper JMX enabled by default Using config: /opt/bitnami/zookeeper/bin/../conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Error contacting service. It is probably not running.

From the above output, you can see the ZooKeeper service is not functioning properly. Let us check the cluster configuration for one of the ZooKeeper pods:

$ istioctl proxy-config cluster my-release-zookeeper-1 --port 3888 --direction inbound -o json
[ { "name": "inbound|3888||", "type": "STATIC", "connectTimeout": "10s", "loadAssignment": { "clusterName": "inbound|3888||", "endpoints": [ { "lbEndpoints": [ { "endpoint": { "address": { "socketAddress": { "address": "127.0.0.1", "portValue": 3888 } } } } ] } ] }, ...

What is interesting here is that the inbound on port 3888 has 127.0.0.1 as its endpoint. This is because the Envoy proxy, in versions of Istio prior to 1.10, redirects the inbound traffic to the loopback interface, as described in our blog post about the change.

StatefulSets in action with Istio 1.10

Now, we have upgraded our cluster to Istio 1.10 and configured the default namespace to enable 1.10 sidecar injection. Let’s rolling restart the ZooKeeper StatefulSet to update the pods to use the new version of the sidecar proxy:

$ kubectl rollout restart statefulset my-release-zookeeper

Once the ZooKeeper pods reach the running status, let’s check out the network connections for these 3 ports from any of the ZooKeeper pods:

$ kubectl exec my-release-zookeeper-1 -c istio-proxy -- netstat -na | grep -E '(2181|2888|3888)'
tcp 0 0 0.0.0.0:2181 0.0.0.0:* LISTEN tcp 0 0 10.96.8.10:2888 0.0.0.0:* LISTEN tcp 0 0 10.96.8.10:3888 0.0.0.0:* LISTEN tcp 0 0 127.0.0.6:42571 10.96.8.10:2888 ESTABLISHED tcp 0 0 10.96.8.10:2888 127.0.0.6:42571 ESTABLISHED tcp 0 0 127.0.0.6:42655 10.96.8.10:2888 ESTABLISHED tcp 0 0 10.96.8.10:2888 127.0.0.6:42655 ESTABLISHED tcp 0 0 10.96.8.10:37876 10.96.6.11:3888 ESTABLISHED tcp 0 0 10.96.8.10:44872 10.96.7.10:3888 ESTABLISHED tcp 0 0 10.96.8.10:37878 10.96.6.11:3888 ESTABLISHED tcp 0 0 10.96.8.10:44870 10.96.7.10:3888 ESTABLISHED tcp 0 0 127.0.0.1:2181 127.0.0.1:54508 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:54616 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:54664 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:54526 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:54532 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:54578 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:54634 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:54588 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:54610 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:54550 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:54560 TIME_WAIT tcp 0 0 127.0.0.1:2181 127.0.0.1:54644 TIME_WAIT

There are ESTABLISHED connections on both port 2888 and 3888! Next, let us check out the ZooKeeper server status:

$ kubectl exec my-release-zookeeper-1 -c zookeeper -- /opt/bitnami/zookeeper/bin/zkServer.sh status
/opt/bitnami/java/bin/java ZooKeeper JMX enabled by default Using config: /opt/bitnami/zookeeper/bin/../conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Mode: follower

The ZooKeeper service is now running!

We can connect to each of the ZooKeeper pods from the sleep pod and run the below command to discover the server status of each pod within the StatefulSet. Note that there is no need to create ServiceEntry resources for any of the ZooKeeper pods and we can call these pods directly using their DNS names (e.g. my-release-zookeeper-0.my-release-zookeeper-headless) from the sleep pod.

$ kubectl exec -it deploy/sleep -c sleep -- sh -c 'for x in my-release-zookeeper-0.my-release-zookeeper-headless my-release-zookeeper-1.my-release-zookeeper-headless my-release-zookeeper-2.my-release-zookeeper-headless; do echo $x; echo srvr|nc $x 2181; echo; done'
my-release-zookeeper-0.my-release-zookeeper-headless Zookeeper version: 3.7.0-e3704b390a6697bfdf4b0bef79e3da7a4f6bac4b, built on 2021-03-17 09:46 UTC Latency min/avg/max: 1/7.5/20 Received: 3845 Sent: 3844 Connections: 1 Outstanding: 0 Zxid: 0x200000002 Mode: follower Node count: 6 my-release-zookeeper-1.my-release-zookeeper-headless Zookeeper version: 3.7.0-e3704b390a6697bfdf4b0bef79e3da7a4f6bac4b, built on 2021-03-17 09:46 UTC Latency min/avg/max: 0/0.0/0 Received: 3856 Sent: 3855 Connections: 1 Outstanding: 0 Zxid: 0x200000002 Mode: follower Node count: 6 my-release-zookeeper-2.my-release-zookeeper-headless Zookeeper version: 3.7.0-e3704b390a6697bfdf4b0bef79e3da7a4f6bac4b, built on 2021-03-17 09:46 UTC Latency min/avg/max: 0/0.0/0 Received: 3855 Sent: 3854 Connections: 1 Outstanding: 0 Zxid: 0x200000002 Mode: leader Node count: 6 Proposal sizes last/min/max: 48/48/48

Now our ZooKeeper service is running, let’s use Istio to secure all communication to our regular and headless services. Apply mutual TLS to the default namespace:

$ kubectl apply -n default -f - <<EOF apiVersion: "security.istio.io/v1beta1" kind: "PeerAuthentication" metadata: name: "default" spec: mtls: mode: STRICT EOF

Continue sending some traffic from the sleep pod and bring up the Kiali dashboard to visualize the services in the default namespace:

Visualize the ZooKeeper Services in Kiali
Visualize the ZooKeeper Services in Kiali

The padlock icons on the traffic flows indicate that the connections are secure.

Wrapping up

With the new networking changes in Istio 1.10, a Kubernetes pod with a sidecar has the same networking behavior as a pod without a sidecar. This change enables stateful applications to function properly in Istio as we have shown you in this post. We believe this is a huge step towards Istio’s goal of providing transparent service mesh and zero-configuration Istio.

Share this post