StatefulSets Made Easier With Istio 1.10
Learn how to easily deploy StatefulSets with Istio 1.10.
Kubernetes StatefulSets
are commonly used to manage stateful applications. In addition to managing the deployment and scaling of a set of Pods
, StatefulSets
provide guarantees about the ordering and uniqueness of those Pods
. Common applications used with StatefulSets
include ZooKeeper, Cassandra, Elasticsearch, Redis and NiFi.
The Istio community has been making gradual progress towards zero-configuration support for StatefulSets
; from automatic mTLS, to eliminating the need to create DestinationRule
or ServiceEntry
resources, to the most recent pod networking changes in Istio 1.10.
What is unique about using a StatefulSet
with a service mesh? The StatefulSet
pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling. The kind of apps that run in a StatefulSet
are often those that need to communicate among their pods, and, as they come from a world of hard-coded IP addresses, may listen on the pod IP only, instead of 0.0.0.0
.
ZooKeeper, for example, is configured by default to not listen on all IPs for quorum communication:
Over the last few releases, the Istio community has reported many issues around support for applications running in StatefulSets
.
StatefulSets
in action, prior to Istio 1.10
In a GKE cluster running Kubernetes 1.19, we have Istio 1.9.5 installed. We enabled automatic sidecar injection in the default
namespace, then we installed ZooKeeper using the Helm charts provided by Bitnami, along with the Istio sleep
pod for interactive debugging:
After a few minutes, all pods come up nicely with sidecar proxies:
Are our ZooKeeper services working and is the status Running
? Let’s find out! ZooKeeper listens on 3 ports:
- Port 2181 is the TCP port for clients to connect to the ZooKeeper service
- Port 2888 is the TCP port for peers to connect to other peers
- Port 3888 is the dedicated TCP port for leader election
By default, the ZooKeeper installation configures port 2181 to listen on 0.0.0.0
but ports 2888 and 3888 only listen on the pod IP. Let’s check out the network status on each of these ports from one of the ZooKeeper pods:
There is nothing ESTABLISHED
on port 2888 or 3888. Next, let us get the ZooKeeper server status:
From the above output, you can see the ZooKeeper service is not functioning properly. Let us check the cluster configuration for one of the ZooKeeper pods:
What is interesting here is that the inbound on port 3888 has 127.0.0.1
as its endpoint. This is because the Envoy proxy, in versions of Istio prior to 1.10, redirects the inbound traffic to the loopback
interface, as described in our blog post about the change.
StatefulSets
in action with Istio 1.10
Now, we have upgraded our cluster to Istio 1.10 and configured the default
namespace to enable 1.10 sidecar injection. Let’s rolling restart the ZooKeeper StatefulSet
to update the pods to use the new version of the sidecar proxy:
Once the ZooKeeper pods reach the running status, let’s check out the network connections for these 3 ports from any of the ZooKeeper pods:
There are ESTABLISHED
connections on both port 2888 and 3888! Next, let us check out the ZooKeeper server status:
The ZooKeeper service is now running!
We can connect to each of the ZooKeeper pods from the sleep
pod and run the below command to discover the server status of each pod within the StatefulSet
. Note that there is no need to create ServiceEntry resources for any of the ZooKeeper pods and we can call these pods directly using their DNS names (e.g. my-release-zookeeper-0.my-release-zookeeper-headless
) from the sleep
pod.
Now our ZooKeeper service is running, let’s use Istio to secure all communication to our regular and headless services. Apply mutual TLS to the default
namespace:
Continue sending some traffic from the sleep
pod and bring up the Kiali dashboard to visualize the services in the default
namespace:
The padlock icons on the traffic flows indicate that the connections are secure.
Wrapping up
With the new networking changes in Istio 1.10, a Kubernetes pod with a sidecar has the same networking behavior as a pod without a sidecar. This change enables stateful applications to function properly in Istio as we have shown you in this post. We believe this is a huge step towards Istio’s goal of providing transparent service mesh and zero-configuration Istio.