Canary Upgrades
Upgrading Istio can be done by first running a canary deployment of the new control plane, allowing you to monitor the effect of the upgrade with a small percentage of the workloads before migrating all of the traffic to the new version. This is much safer than doing an in-place upgrade and is the recommended upgrade method.
When installing Istio, the revision
installation setting can be used to deploy multiple independent control planes
at the same time. A canary version of an upgrade can be started by installing the new Istio version’s control plane
next to the old one, using a different revision
setting. Each revision is a full Istio control plane implementation
with its own Deployment
, Service
, etc.
Before you upgrade
Before upgrading Istio, it is recommended to run the istioctl x precheck
command to make sure the upgrade is compatible with your environment.
$ istioctl x precheck
✔ No issues found when checking the cluster. Istio is safe to install or upgrade!
To get started, check out https://istio.io/latest/docs/setup/getting-started/
Control plane
To install a new revision called canary
, you would set the revision
field as follows:
$ istioctl install --set revision=canary
After running the command, you will have two control plane deployments and services running side-by-side:
$ kubectl get pods -n istio-system -l app=istiod
NAME READY STATUS RESTARTS AGE
istiod-1-26-1-bdf5948d5-htddg 1/1 Running 0 47s
istiod-canary-84c8d4dcfb-skcfv 1/1 Running 0 25s
$ kubectl get svc -n istio-system -l app=istiod
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istiod-1-26-1 ClusterIP 10.96.93.151 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP 109s
istiod-canary ClusterIP 10.104.186.250 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP 87s
You will also see that there are two sidecar injector configurations including the new revision.
$ kubectl get mutatingwebhookconfigurations
NAME WEBHOOKS AGE
istio-sidecar-injector-1-26-1 2 2m16s
istio-sidecar-injector-canary 2 114s
data plane
Refer to Gateway Canary Upgrade to understand how to run revision specific instances of Istio gateway.
In this example, since we use the default
Istio profile, Istio gateways do not run revision-specific instances, but are instead in-place upgraded to use the new control plane revision. You can verify that the istio-ingress
gateway is using the canary
revision by running the following command:
$ istioctl proxy-status | grep "$(kubectl -n istio-system get pod -l app=istio-ingressgateway -o jsonpath='{.items..metadata.name}')" | awk -F '[[:space:]][[:space:]]+' '{print $8}'
istiod-canary-6956db645c-vwhsk
However, simply installing the new revision has no impact on the existing sidecar proxies. To upgrade these,
you must configure them to point to the new istiod-canary
control plane. This is controlled during sidecar injection
based on the namespace label istio.io/rev
.
Create a namespace test-ns
with istio-injection
enabled. In the test-ns
namespace, deploy a sample curl pod:
Create a namespace
test-ns
.$ kubectl create ns test-ns
Label the namespace using
istio-injection
label.$ kubectl label namespace test-ns istio-injection=enabled
Bring up a sample curl pod in
test-ns
namespace.$ kubectl apply -n test-ns -f samples/curl/curl.yaml
To upgrade the namespace test-ns
, remove the istio-injection
label, and add the istio.io/rev
label to point to the canary
revision. The istio-injection
label must be removed because it takes precedence over the istio.io/rev
label for backward compatibility.
$ kubectl label namespace test-ns istio-injection- istio.io/rev=canary
After the namespace updates, you need to restart the pods to trigger re-injection.
One way to restart all pods in namespace test-ns
is using:
$ kubectl rollout restart deployment -n test-ns
When the pods are re-injected, they will be configured to point to the istiod-canary
control plane. You can verify this by using istioctl proxy-status
.
$ istioctl proxy-status | grep "\.test-ns "
The output will show all pods under the namespace that are using the canary revision.
Stable revision labels
Reetiquetar manualmente los namespaces al moverlos a una nueva revisión puede ser tedioso y propenso a errores. Las etiquetas de revisión resuelven este problema. Las etiquetas de revisión son identificadores estables que apuntan a revisiones y se pueden usar para evitar reetiquetar namespaces. En lugar de reetiquetar el namespace, un operador de malla puede simplemente cambiar la etiqueta para que apunte a una nueva revisión. Todos los namespaces etiquetados con esa etiqueta se actualizarán al mismo tiempo.Usage
Considera un cluster con dos revisiones instaladas,1-26-1
y 1-27-0
. El operador del cluster crea una etiqueta de revisión prod-stable
,
apuntando a la versión más antigua y estable 1-26-1
, y una etiqueta de revisión prod-canary
apuntando a la revisión más nueva 1-27-0
. Ese
estado podría alcanzarse a través de los siguientes comandos:Install two revisions of control plane:
$ istioctl install --revision=1-26-1 --set profile=minimal --skip-confirmation $ istioctl install --revision=1-27-0 --set profile=minimal --skip-confirmation
Create
stable
andcanary
revision tags and associate them to the respective revisions:$ istioctl tag set prod-stable --revision 1-26-1 $ istioctl tag set prod-canary --revision 1-27-0
Label application namespaces to map to the respective revision tags:
$ kubectl create ns app-ns-1 $ kubectl label ns app-ns-1 istio.io/rev=prod-stable $ kubectl create ns app-ns-2 $ kubectl label ns app-ns-2 istio.io/rev=prod-stable $ kubectl create ns app-ns-3 $ kubectl label ns app-ns-3 istio.io/rev=prod-canary
Bring up a sample curl pod in each namespace:
$ kubectl apply -n app-ns-1 -f samples/curl/curl.yaml $ kubectl apply -n app-ns-2 -f samples/curl/curl.yaml $ kubectl apply -n app-ns-3 -f samples/curl/curl.yaml
Verify application to control plane mapping using
istioctl proxy-status
command:$ istioctl ps NAME CLUSTER CDS LDS EDS RDS ECDS ISTIOD VERSION curl-78ff5975c6-62pzf.app-ns-3 Kubernetes SYNCED SYNCED SYNCED SYNCED NOT SENT istiod-1-27-0-7f6fc6cfd6-s8zfg 1.27.0 curl-78ff5975c6-8kxpl.app-ns-1 Kubernetes SYNCED SYNCED SYNCED SYNCED NOT SENT istiod-1-26-1-bdf5948d5-n72r2 1.26.1 curl-78ff5975c6-8q7m6.app-ns-2 Kubernetes SYNCED SYNCED SYNCED SYNCED NOT SENT istiod-1-26-1-bdf5948d5-n72r2 1-26.1
La asignación resultante entre revisiones, etiquetas y namespaces es como se muestra a continuación:
El operador del cluster puede ver esta asignación además de los namespaces etiquetados a través del comando istioctl tag list
:
$ istioctl tag list
TAG REVISION NAMESPACES
default 1-26-1 ...
prod-canary 1-27-0 ...
prod-stable 1-26-1 ...
Después de que el operador del cluster esté satisfecho con la estabilidad del control plane etiquetado con prod-canary
, los namespaces etiquetados
istio.io/rev=prod-stable
se pueden actualizar con una acción modificando la etiqueta de revisión prod-stable
para que apunte a la revisión
1-27-0
más nueva.
$ istioctl tag set prod-stable --revision 1-27-0 --overwrite
Ahora, la asignación actualizada entre revisiones, etiquetas y namespaces es como se muestra a continuación:
Reiniciar los workloads inyectadas en los namespaces marcados como prod-stable
ahora dará como resultado que esas workloads usen el
control plane 1-27-0
.
Observa que no se requirió ningún reetiquetado de namespace para migrar los workloads a la nueva revisión.
$ kubectl rollout restart deployment -n app-ns-1
$ kubectl rollout restart deployment -n app-ns-2
Verify the application to control plane mapping using istioctl proxy-status
command:
$ istioctl ps
NAME CLUSTER CDS LDS EDS RDS ECDS ISTIOD VERSION
curl-5984f48bc7-kmj6x.app-ns-1 Kubernetes SYNCED SYNCED SYNCED SYNCED NOT SENT istiod-1-27-0-7f6fc6cfd6-jsktb 1.27.0
curl-78ff5975c6-jldk4.app-ns-3 Kubernetes SYNCED SYNCED SYNCED SYNCED NOT SENT istiod-1-27-0-7f6fc6cfd6-jsktb 1.27.0
curl-7cdd8dccb9-5bq5n.app-ns-2 Kubernetes SYNCED SYNCED SYNCED SYNCED NOT SENT istiod-1-27-0-7f6fc6cfd6-jsktb 1.27.0
Default tag
La revisión a la que apunta la etiqueta default
se considera la revisión predeterminada y tiene un significado semántico adicional. La revisión predeterminada
realiza las siguientes funciones:
- Inyecta sidecars para el selector de namespace
istio-injection=enabled
, el selector de objetossidecar.istio.io/inject=true
y los selectoresistio.io/rev=default
- Valida los recursos de Istio
- Roba el bloqueo de líder de las revisiones no predeterminadas y realiza responsabilidades de malla singleton (como actualizar los estados de los recursos)
Para hacer que una revisión 1-27-0
sea la predeterminada, ejecuta:
$ istioctl tag set default --revision 1-27-0
Cuando se utiliza la etiqueta default
junto con una instalación de Istio existente sin revisión, se recomienda eliminar la
MutatingWebhookConfiguration
antigua (normalmente llamada istio-sidecar-injector
) para evitar que tanto el
control plane antiguo como el más nuevo intenten la inyección.Uninstall old control plane
After upgrading both the control plane and data plane, you can uninstall the old control plane. For example, the following command uninstalls a control plane of revision 1-26-1
:
$ istioctl uninstall --revision 1-26-1 -y
If the old control plane does not have a revision label, uninstall it using its original installation options, for example:
$ istioctl uninstall -f manifests/profiles/default.yaml -y
Confirm that the old control plane has been removed and only the new one still exists in the cluster:
$ kubectl get pods -n istio-system -l app=istiod
NAME READY STATUS RESTARTS AGE
istiod-canary-55887f699c-t8bh8 1/1 Running 0 27m
Note that the above instructions only removed the resources for the specified control plane revision, but not cluster-scoped resources shared with other control planes. To uninstall Istio completely, refer to the uninstall guide.
Uninstall canary control plane
If you decide to rollback to the old control plane, instead of completing the canary upgrade, you can uninstall the canary revision using:
$ istioctl uninstall --revision=canary -y
However, in this case you must first reinstall the gateway(s) for the previous revision manually, because the uninstall command will not automatically revert the previously in-place upgraded ones.
Cleanup
Clean up created revisioned tags:
$ istioctl tag remove prod-stable $ istioctl tag remove prod-canary
Clean up the namespaces used for canary upgrade with revision labels example:
$ kubectl delete ns istio-system test-ns
Clean up the namespaces used for canary upgrade with revision tags example:
$ kubectl delete ns istio-system app-ns-1 app-ns-2 app-ns-3