Mirroring

This task demonstrates the traffic mirroring capabilities of Istio.

Traffic mirroring, also called shadowing, is a powerful concept that allows feature teams to bring changes to production with as little risk as possible. Mirroring sends a copy of live traffic to a mirrored service. The mirrored traffic happens out of band of the critical request path for the primary service.

In this task, you will first force all traffic to v1 of a test service. Then, you will apply a rule to mirror a portion of traffic to v2.

Before you begin

1. Set up Istio by following the Installation guide⁶.

2. Start by deploying two versions of the httpbin⁷ service that have access logging enabled:

   1. Deploy httpbin-v1:

      $ kubectl create -f - <<EOF
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: httpbin-v1
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: httpbin
            version: v1
        template:
          metadata:
            labels:
              app: httpbin
              version: v1
          spec:
            containers:
            - image: docker.io/kennethreitz/httpbin
              imagePullPolicy: IfNotPresent
              name: httpbin
              command: ["gunicorn", "--access-logfile", "-", "-b", "0.0.0.0:80", "httpbin:app"]
              ports:
              - containerPort: 80
      EOF
      

   2. Deploy httpbin-v2:

      $ kubectl create -f - <<EOF
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: httpbin-v2
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: httpbin
            version: v2
        template:
          metadata:
            labels:
              app: httpbin
              version: v2
          spec:
            containers:
            - image: docker.io/kennethreitz/httpbin
              imagePullPolicy: IfNotPresent
              name: httpbin
              command: ["gunicorn", "--access-logfile", "-", "-b", "0.0.0.0:80", "httpbin:app"]
              ports:
              - containerPort: 80
      EOF
      

   3. Deploy the httpbin Kubernetes service:

      $ kubectl create -f - <<EOF
      apiVersion: v1
      kind: Service
      metadata:
        name: httpbin
        labels:
          app: httpbin
      spec:
        ports:
        - name: http
          port: 8000
          targetPort: 80
        selector:
          app: httpbin
      EOF
      

3. Deploy the curl workload you’ll use to send requests to the httpbin service:

   $ cat <<EOF | kubectl create -f -
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: curl
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: curl
     template:
       metadata:
         labels:
           app: curl
       spec:
         containers:
         - name: curl
           image: curlimages/curl
           command: ["/bin/sleep","3650d"]
           imagePullPolicy: IfNotPresent
   EOF
   

Creating a default routing policy

By default Kubernetes load balances across both versions of the httpbin service. In this step, you will change that behavior so that all traffic goes to v1.

1. Create a default route rule to route all traffic to v1 of the service:

   Istio APIs Gateway API

   $ kubectl apply -f - <<EOF
   apiVersion: networking.istio.io/v1
   kind: VirtualService
   metadata:
     name: httpbin
   spec:
     hosts:
   - httpbin
     http:
     - route:
   - destination:
       host: httpbin
       subset: v1
     weight: 100
   ---
   apiVersion: networking.istio.io/v1
   kind: DestinationRule
   metadata:
     name: httpbin
   spec:
     host: httpbin
     subsets:
     - name: v1
   labels:
     version: v1
     - name: v2
   labels:
     version: v2
   EOF
   

2. Now, with all traffic directed to httpbin:v1, send a request to the service:

   $ kubectl exec deploy/curl -c curl -- curl -sS http://httpbin:8000/headers
   {
     "headers": {
       "Accept": "*/*",
       "Content-Length": "0",
       "Host": "httpbin:8000",
       "User-Agent": "curl/7.35.0",
       "X-B3-Parentspanid": "57784f8bff90ae0b",
       "X-B3-Sampled": "1",
       "X-B3-Spanid": "3289ae7257c3f159",
       "X-B3-Traceid": "b56eebd279a76f0b57784f8bff90ae0b",
       "X-Envoy-Attempt-Count": "1",
       "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/default/sa/default;Hash=20afebed6da091c850264cc751b8c9306abac02993f80bdb76282237422bd098;Subject=\"\";URI=spiffe://cluster.local/ns/default/sa/default"
     }
   }

3. Check the logs from httpbin-v1 and httpbin-v2 pods. You should see access log entries for v1 and none for v2:

   $ kubectl logs deploy/httpbin-v1 -c httpbin
   127.0.0.1 - - [07/Mar/2018:19:02:43 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"

   $ kubectl logs deploy/httpbin-v2 -c httpbin
   <none>

Mirroring traffic to httpbin-v2

1. Change the route rule to mirror traffic to httpbin-v2:

   Istio APIs Gateway API

   $ kubectl apply -f - <<EOF
   apiVersion: networking.istio.io/v1
   kind: VirtualService
   metadata:
     name: httpbin
   spec:
     hosts:
   - httpbin
     http:
     - route:
   - destination:
       host: httpbin
       subset: v1
     weight: 100
   mirror:
     host: httpbin
     subset: v2
   mirrorPercentage:
     value: 100.0
   EOF
   

   This route rule sends 100% of the traffic to v1. The last stanza specifies that you want to mirror (i.e., also send) 100% of the same traffic to the httpbin:v2 service. When traffic gets mirrored, the requests are sent to the mirrored service with their Host/Authority headers appended with -shadow. For example, cluster-1 becomes cluster-1-shadow.

   Also, it is important to note that these requests are mirrored as “fire and forget”, which means that the responses are discarded.

   You can use the value field under the mirrorPercentage field to mirror a fraction of the traffic, instead of mirroring all requests. If this field is absent, all traffic will be mirrored.

2. Send the traffic:

   $ kubectl exec deploy/curl -c curl -- curl -sS http://httpbin:8000/headers
   

   Now, you should see access logging for both v1 and v2. The access logs created in v2 are the mirrored requests that are actually going to v1.

   $ kubectl logs deploy/httpbin-v1 -c httpbin
   127.0.0.1 - - [07/Mar/2018:19:02:43 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"
   127.0.0.1 - - [07/Mar/2018:19:26:44 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"

   $ kubectl logs deploy/httpbin-v2 -c httpbin
   127.0.0.1 - - [07/Mar/2018:19:26:44 +0000] "GET /headers HTTP/1.1" 200 361 "-" "curl/7.35.0"

Cleaning up

1. Remove the rules:

   Istio APIs Gateway API

   $ kubectl delete virtualservice httpbin
   $ kubectl delete destinationrule httpbin
   

2. Delete httpbin and curl deployments and httpbin service:

   $ kubectl delete deploy httpbin-v1 httpbin-v2 curl
   $ kubectl delete svc httpbin