Install Multi-Primary on different networks

 5 minute read     page test

Follow this guide to install the Istio control plane on both cluster1 and cluster2, making each a primary cluster. Cluster cluster1 is on the network1 network, while cluster2 is on the network2 network. This means there is no direct connectivity between pods across cluster boundaries.

Before proceeding, be sure to complete the steps under before you begin.

In this configuration, both cluster1 and cluster2 observe the API Servers in each cluster for endpoints.

Service workloads across cluster boundaries communicate indirectly, via dedicated gateways for east-west traffic. The gateway in each cluster must be reachable from the other cluster.

Multiple primary clusters on separate networks
Multiple primary clusters on separate networks

Set the default network for cluster1

If the istio-system namespace is already created, we need to set the cluster’s network there:

$ kubectl --context="${CTX_CLUSTER1}" get namespace istio-system && \
  kubectl --context="${CTX_CLUSTER1}" label namespace istio-system topology.istio.io/network=network1

Configure cluster1 as a primary

Create the istioctl configuration for cluster1:

Install Istio as primary in cluster1 using istioctl and the IstioOperator API.

$ cat <<EOF > cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster1
      network: network1
EOF

Apply the configuration to cluster1:

$ istioctl install --context="${CTX_CLUSTER1}" -f cluster1.yaml

Install the east-west gateway in cluster1

Install a gateway in cluster1 that is dedicated to east-west traffic. By default, this gateway will be public on the Internet. Production systems may require additional access restrictions (e.g. via firewall rules) to prevent external attacks. Check with your cloud vendor to see what options are available.

Zip
$ @samples/multicluster/gen-eastwest-gateway.sh@ \
    --network network1 | \
    istioctl --context="${CTX_CLUSTER1}" install -y -f -

Wait for the east-west gateway to be assigned an external IP address:

$ kubectl --context="${CTX_CLUSTER1}" get svc istio-eastwestgateway -n istio-system
NAME                    TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)   AGE
istio-eastwestgateway   LoadBalancer   10.80.6.124   34.75.71.237   ...       51s

Expose services in cluster1

Since the clusters are on separate networks, we need to expose all services (*.local) on the east-west gateway in both clusters. While this gateway is public on the Internet, services behind it can only be accessed by services with a trusted mTLS certificate and workload ID, just as if they were on the same network.

Zip
$ kubectl --context="${CTX_CLUSTER1}" apply -n istio-system -f \
    @samples/multicluster/expose-services.yaml@

Set the default network for cluster2

If the istio-system namespace is already created, we need to set the cluster’s network there:

$ kubectl --context="${CTX_CLUSTER2}" get namespace istio-system && \
  kubectl --context="${CTX_CLUSTER2}" label namespace istio-system topology.istio.io/network=network2

Configure cluster2 as a primary

Create the istioctl configuration for cluster2:

Install Istio as primary in cluster2 using istioctl and the IstioOperator API.

$ cat <<EOF > cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster2
      network: network2
EOF

Apply the configuration to cluster2:

$ istioctl install --context="${CTX_CLUSTER2}" -f cluster2.yaml

Install the east-west gateway in cluster2

As we did with cluster1 above, install a gateway in cluster2 that is dedicated to east-west traffic.

Zip
$ @samples/multicluster/gen-eastwest-gateway.sh@ \
    --network network2 | \
    istioctl --context="${CTX_CLUSTER2}" install -y -f -

Wait for the east-west gateway to be assigned an external IP address:

$ kubectl --context="${CTX_CLUSTER2}" get svc istio-eastwestgateway -n istio-system
NAME                    TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)   AGE
istio-eastwestgateway   LoadBalancer   10.0.12.121   34.122.91.98   ...       51s

Expose services in cluster2

As we did with cluster1 above, expose services via the east-west gateway.

Zip
$ kubectl --context="${CTX_CLUSTER2}" apply -n istio-system -f \
    @samples/multicluster/expose-services.yaml@

Enable Endpoint Discovery

Install a remote secret in cluster2 that provides access to cluster1’s API server.

$ istioctl create-remote-secret \
  --context="${CTX_CLUSTER1}" \
  --name=cluster1 | \
  kubectl apply -f - --context="${CTX_CLUSTER2}"

Install a remote secret in cluster1 that provides access to cluster2’s API server.

$ istioctl create-remote-secret \
  --context="${CTX_CLUSTER2}" \
  --name=cluster2 | \
  kubectl apply -f - --context="${CTX_CLUSTER1}"

Congratulations! You successfully installed an Istio mesh across multiple primary clusters on different networks!

Next Steps

You can now verify the installation.

Cleanup

Uninstall Istio from both cluster1 and cluster2 using the same mechanism you installed Istio with (istioctl or Helm).

Uninstall Istio in cluster1:

$ istioctl uninstall --context="${CTX_CLUSTER1}" -y --purge
$ kubectl delete ns istio-system --context="${CTX_CLUSTER1}"

Uninstall Istio in cluster2:

$ istioctl uninstall --context="${CTX_CLUSTER2}" -y --purge
$ kubectl delete ns istio-system --context="${CTX_CLUSTER2}"
Was this information useful?
Do you have any suggestions for improvement?

Thanks for your feedback!