Control Egress Traffic

Because all outbound traffic from an Istio-enabled pod is redirected to its sidecar proxy by default, Istio must be configured in a way that allows access to URLs outside of the cluster before external services can be called. This task will show you how to do it in three different ways:

  1. Configure the Envoy proxy to pass requests through to external services on ports that are not configured inside the mesh.
  2. Configure service entries to provide controlled access to external services.
  3. Bypass the Envoy proxy for a specific range of IPs.

Before you begin

  • Setup Istio by following the instructions in the Installation guide.

  • To have test source for sending requests, start the sleep sample.

    If you have enabled automatic sidecar injection, do

    Zip
    $ kubectl apply -f @samples/sleep/sleep.yaml@
    

    otherwise, you have to manually inject the sidecar before deploying the sleep application:

    Zip
    $ kubectl apply -f <(istioctl kube-inject -f @samples/sleep/sleep.yaml@)
    

    You can use any pod with curl installed as a test source.

  • To send requests, create the SOURCE_POD environment variable to store the name of the source pod:

    $ export SOURCE_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
    

Envoy passthrough to external services

Istio has an installation option that configures the Istio proxy to pass through, instead of block, calls to external services on any ports without an HTTP service or service entry within the mesh. This option, global.outboundTrafficPolicy.mode=ALLOW_ANY, is enabled by default so that one can start evaluating Istio quickly, without controlling access to external services, and decide to configure access to external services later.

For example, if you don’t register an HTTP service or define a service entry for port 8000 within the mesh, the sidecar proxy will forward requests to any external service on that port. However, if you later create an HTTP service inside the mesh on port 8000 or define a service entry for any host on port 8000, Istio will block all external access to port 8000; Istio will fall back to a blocking-by-default behavior for that port.

  1. To demonstrate this approach you need to ensure that your Istio installation is configured with the global.outboundTrafficPolicy.mode option set to ALLOW_ANY. Unless you explicitly disabled it using the --set global.outboundTrafficPolicy.mode=REGISTRY_ONLY flag when you installed Istio using helm, it is enabled by default.

  2. Make a couple of requests to external HTTPS services from SOURCE_POD:

    $ kubectl exec -it $SOURCE_POD -c sleep -- curl -s https://www.google.com | grep -o "<title>.*</title>"; kubectl exec -it $SOURCE_POD -c sleep -- curl -s https://edition.cnn.com | grep -o "<title>.*</title>"
    <title>Google</title>
    <title>CNN International - Breaking News, US News, World News and Video</title>
    

Note that if you configure Istio with allowed access to external services this way, you lose Istio monitoring on traffic to external services: the calls to external services will not appear in the Mixer log, for example. To start monitoring access to external services, follow the steps in configure access to external services (no need to update Istio).

Controlled access to external services

Using Istio ServiceEntry configurations, you can access any publicly accessible service from within your Istio cluster. This task shows you how to access an external HTTP service, httpbin.org, as well as an external HTTPS service, www.google.com.

Change to the blocking-by-default policy

Before you can experiment with a controlled way of accessing external services, you need to disable the passthrough mode used in the previous section:

  1. Run the following command to change the global.outboundTrafficPolicy.mode option to REGISTRY_ONLY:

    $ kubectl get configmap istio -n istio-system --export -o yaml | sed 's/mode: ALLOW_ANY/mode: REGISTRY_ONLY/g' | kubectl replace -n istio-system -f -
    configmap "istio" replaced
    
  2. Make a couple of requests to external HTTPS services from SOURCE_POD to verify that they are now blocked:

    $ kubectl exec -it $SOURCE_POD -c sleep -- curl -s https://www.google.com | grep -o "<title>.*</title>"; kubectl exec -it $SOURCE_POD -c sleep -- curl -s https://edition.cnn.com | grep -o "<title>.*</title>"
    command terminated with exit code 35
    command terminated with exit code 35
    

Access an external HTTP service

  1. Create a ServiceEntry to allow access to an external HTTP service:

    $ kubectl apply -f - <<EOF
    apiVersion: networking.istio.io/v1alpha3
    kind: ServiceEntry
    metadata:
      name: httpbin-ext
    spec:
      hosts:
      - httpbin.org
      ports:
      - number: 80
        name: http
        protocol: HTTP
      resolution: DNS
      location: MESH_EXTERNAL
    EOF
    
  2. Make a request to the external HTTP service from SOURCE_POD:

    $  kubectl exec -it $SOURCE_POD -c sleep -- curl http://httpbin.org/headers
    {
      "headers": {
      "Accept": "*/*",
      "Connection": "close",
      "Host": "httpbin.org",
      "User-Agent": "curl/7.60.0",
      ...
      "X-Envoy-Decorator-Operation": "httpbin.org:80/*",
      }
    }
    

    Note the headers added by the Istio sidecar proxy: X-Envoy-Decorator-Operation.

  3. Check the log of the sidecar proxy of SOURCE_POD:

    $  kubectl logs $SOURCE_POD -c istio-proxy | tail
    [2019-01-24T12:17:11.640Z] "GET /headers HTTP/1.1" 200 - 0 599 214 214 "-" "curl/7.60.0" "17fde8f7-fa62-9b39-8999-302324e6def2" "httpbin.org" "35.173.6.94:80" outbound|80||httpbin.org - 35.173.6.94:80 172.30.109.82:55314 -
    

    Note the entry related to your HTTP request to httpbin.org/headers.

  4. Check the Mixer log. If Istio is deployed in the istio-system namespace, the command to print the log is:

    $ kubectl -n istio-system logs -l istio-mixer-type=telemetry -c mixer | grep 'httpbin.org'
    {"level":"info","time":"2019-01-24T12:17:11.855496Z","instance":"accesslog.logentry.istio-system","apiClaims":"","apiKey":"","clientTraceId":"","connection_security_policy":"unknown","destinationApp":"","destinationIp":"I60GXg==","destinationName":"unknown","destinationNamespace":"default","destinationOwner":"unknown","destinationPrincipal":"","destinationServiceHost":"httpbin.org","destinationWorkload":"unknown","grpcMessage":"","grpcStatus":"","httpAuthority":"httpbin.org","latency":"214.661667ms","method":"GET","permissiveResponseCode":"none","permissiveResponsePolicyID":"none","protocol":"http","receivedBytes":270,"referer":"","reporter":"source","requestId":"17fde8f7-fa62-9b39-8999-302324e6def2","requestSize":0,"requestedServerName":"","responseCode":200,"responseSize":599,"responseTimestamp":"2019-01-24T12:17:11.855521Z","sentBytes":806,"sourceApp":"sleep","sourceIp":"AAAAAAAAAAAAAP//rB5tUg==","sourceName":"sleep-88ddbcfdd-rgk77","sourceNamespace":"default","sourceOwner":"kubernetes://apis/apps/v1/namespaces/default/deployments/sleep","sourcePrincipal":"","sourceWorkload":"sleep","url":"/headers","userAgent":"curl/7.60.0","xForwardedFor":"0.0.0.0"}
    

    Note that the destinationServiceHost attribute is equal to httpbin.org. Also notice the HTTP-related attributes: method, url, responseCode and others. Using Istio egress traffic control, you can monitor access to external HTTP services, including the HTTP-related information of each access.

Access an external HTTPS service

  1. Create a ServiceEntry to allow access to an external HTTPS service.

    $ kubectl apply -f - <<EOF
    apiVersion: networking.istio.io/v1alpha3
    kind: ServiceEntry
    metadata:
      name: google
    spec:
      hosts:
      - www.google.com
      ports:
      - number: 443
        name: https
        protocol: HTTPS
      resolution: DNS
      location: MESH_EXTERNAL
    EOF
    
  2. Make a request to the external HTTPS service from SOURCE_POD:

    $ kubectl exec -it $SOURCE_POD -c sleep -- curl https://www.google.com | grep -o "<title>.*</title>"
    <title>Google</title>
    
  3. Check the log of the sidecar proxy of SOURCE_POD:

    $ kubectl logs $SOURCE_POD -c istio-proxy | tail
    [2019-01-24T12:48:54.977Z] "- - -" 0 - 601 17766 1289 - "-" "-" "-" "-" "172.217.161.36:443" outbound|443||www.google.com 172.30.109.82:59480 172.217.161.36:443 172.30.109.82:59478 www.google.com
    

    Note the entry related to your HTTPS request to www.google.com.

  4. Check the Mixer log. If Istio is deployed in the istio-system namespace, the command to print the log is:

    $ kubectl -n istio-system logs -l istio-mixer-type=telemetry -c mixer | grep 'www.google.com'
    {"level":"info","time":"2019-01-24T12:48:56.266553Z","instance":"tcpaccesslog.logentry.istio-system","connectionDuration":"1.289085134s","connectionEvent":"close","connection_security_policy":"unknown","destinationApp":"","destinationIp":"rNmhJA==","destinationName":"unknown","destinationNamespace":"default","destinationOwner":"unknown","destinationPrincipal":"","destinationServiceHost":"www.google.com","destinationWorkload":"unknown","protocol":"tcp","receivedBytes":601,"reporter":"source","requestedServerName":"www.google.com","sentBytes":17766,"sourceApp":"sleep","sourceIp":"rB5tUg==","sourceName":"sleep-88ddbcfdd-rgk77","sourceNamespace":"default","sourceOwner":"kubernetes://apis/apps/v1/namespaces/default/deployments/sleep","sourcePrincipal":"","sourceWorkload":"sleep","totalReceivedBytes":601,"totalSentBytes":17766}
    

    Note that the requestedServerName attribute is equal to www.google.com. Using Istio egress traffic control, you can monitor access to external HTTPS services, in particular the SNI and the number of sent and received bytes. Note that in HTTPS all the HTTP-related information like method, URL path, response code, is encrypted so Istio cannot see and cannot monitor that information for HTTPS. If you need to monitor HTTP-related information in access to external HTTPS services, you may want to let your applications issue HTTP requests and configure Istio to perform TLS origination.

Manage traffic to external services

Similar to inter-cluster requests, Istio routing rules can also be set for external services that are accessed using ServiceEntry configurations. In this example, you set a timeout rule on calls to the httpbin.org service.

  1. From inside the pod being used as the test source, make a curl request to the /delay endpoint of the httpbin.org external service:

    $ kubectl exec -it $SOURCE_POD -c sleep sh
    $ time curl -o /dev/null -s -w "%{http_code}\n" http://httpbin.org/delay/5
    200
    
    real    0m5.024s
    user    0m0.003s
    sys     0m0.003s
    

    The request should return 200 (OK) in approximately 5 seconds.

  2. Exit the source pod and use kubectl to set a 3s timeout on calls to the httpbin.org external service:

    $ kubectl apply -f - <<EOF
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
      name: httpbin-ext
    spec:
      hosts:
        - httpbin.org
      http:
      - timeout: 3s
        route:
          - destination:
              host: httpbin.org
            weight: 100
    EOF
    
  3. Wait a few seconds, then make the curl request again:

    $ kubectl exec -it $SOURCE_POD -c sleep sh
    $ time curl -o /dev/null -s -w "%{http_code}\n" http://httpbin.org/delay/5
    504
    
    real    0m3.149s
    user    0m0.004s
    sys     0m0.004s
    

    This time a 504 (Gateway Timeout) appears after 3 seconds. Although httpbin.org was waiting 5 seconds, Istio cut off the request at 3 seconds.

Cleanup the controlled access to external services

$ kubectl delete serviceentry httpbin-ext google
$ kubectl delete virtualservice httpbin-ext --ignore-not-found=true

Direct access to external services

If you want to completely bypass Istio for a specific IP range, you can configure the Envoy sidecars to prevent them from intercepting the external requests. This can be done by setting the global.proxy.includeIPRanges variable of Helm and updating the istio-sidecar-injector configmap by using kubectl apply. After istio-sidecar-injector is updated, the value of global.proxy.includeIPRanges will affect all the future deployments of the application pods.

The simplest way to use the global.proxy.includeIPRanges variable is to pass it the IP range(s) used for internal cluster services, thereby excluding external IPs from being redirected to the sidecar proxy. The values used for internal IP range(s), however, depends on where your cluster is running. For example, with Minikube the range is 10.0.0.1/24, so you would update your istio-sidecar-injector configmap like this:

$ helm template install/kubernetes/helm/istio <the flags you used to install Istio> --set global.proxy.includeIPRanges="10.0.0.1/24" -x templates/sidecar-injector-configmap.yaml | kubectl apply -f -

Note that you should use the same Helm command you used to install Istio, in particular, the same value of the --namespace flag. In addition to the flags you used to install Istio, add --set global.proxy.includeIPRanges="10.0.0.1/24" -x templates/sidecar-injector-configmap.yaml.

Redeploy the sleep application as described in the Before you begin section.

Set the value of global.proxy.includeIPRanges

Set the value of global.proxy.includeIPRanges according to your cluster provider.

IBM Cloud Private

  1. Get your service_cluster_ip_range from IBM Cloud Private configuration file under cluster/config.yaml:

    $ cat cluster/config.yaml | grep service_cluster_ip_range
    

    The following is a sample output:

    service_cluster_ip_range: 10.0.0.1/24
    
  2. Use --set global.proxy.includeIPRanges="10.0.0.1/24"

IBM Cloud Kubernetes Service

Use --set global.proxy.includeIPRanges="172.30.0.0/16\,172.21.0.0/16\,10.10.10.0/24"

Google Container Engine (GKE)

The ranges are not fixed, so you will need to run the gcloud container clusters describe command to determine the ranges to use. For example:

$ gcloud container clusters describe XXXXXXX --zone=XXXXXX | grep -e clusterIpv4Cidr -e servicesIpv4Cidr
clusterIpv4Cidr: 10.4.0.0/14
servicesIpv4Cidr: 10.7.240.0/20

Use --set global.proxy.includeIPRanges="10.4.0.0/14\,10.7.240.0/20"

Azure Container Service(ACS)

Use --set global.proxy.includeIPRanges="10.244.0.0/16\,10.240.0.0/16

Minikube, Docker For Desktop, Bare Metal

The default value is 10.96.0.0/12, but it’s not fixed. Use the following command to determine your actual value:

$ kubectl describe pod kube-apiserver -n kube-system | grep 'service-cluster-ip-range'
      --service-cluster-ip-range=10.96.0.0/12

Use --set global.proxy.includeIPRanges="10.96.0.0/12"

Access the external services

After updating the istio-sidecar-injector configmap and redeploying the sleep application, the Istio sidecar will only intercept and manage internal requests within the cluster. Any external request bypasses the sidecar and goes straight to its intended destination. For example:

$ export SOURCE_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
$ kubectl exec -it $SOURCE_POD -c sleep curl http://httpbin.org/headers
{
  "headers": {
    "Accept": "*/*",
    "Connection": "close",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.60.0"
  }
}

Unlike accessing external services through HTTP or HTTPS, you don’t see any headers related to the Istio sidecar and the requests sent to external services appear neither in the log of the sidecar nor in the Mixer log. Bypassing the Istio sidecars means you can no longer monitor the access to external services.

Cleanup the direct access to external services

Update the istio-sidecar-injector.configmap.yaml configuration map to redirect all outbound traffic to the sidecar proxies:

$ helm template install/kubernetes/helm/istio <the flags you used to install Istio> -x templates/sidecar-injector-configmap.yaml | kubectl apply -f -

Understanding what happened

In this task you looked at three ways to call external services from an Istio mesh:

  1. Configuring Istio to allow access to any external service on some ports.

  2. Use a service entry to register an accessible external service inside the mesh. This is the recommended approach.

  3. Configuring the Istio sidecar to exclude external IPs from its remapped IP table.

The first approach directs traffic through the Istio sidecar proxy but it allows access to any service on any port that has no HTTP service or service entry defined in the mesh. You can’t monitor access to external services and you can’t apply Istio features on traffic to external services. You can easily switch to the second approach for a specific port by simply creating a service entry for that port. This means you can use this approach initially to allow access to any external service. Later, you can decide to start controlling access to external services for specific ports and enable traffic monitoring and control features as they are needed.

The second approach lets you use all of the same Istio service mesh features for calls to services inside or outside of the cluster. You saw how to monitor access to external services and set a timeout rule for calls to an external service.

The third approach bypasses the Istio sidecar proxy, giving your services direct access to any external server. However, configuring the proxy this way does require cluster-provider specific knowledge and configuration. Similar to the first approach, you also lose monitoring of access to external services and you can’t apply Istio features on traffic to external services.

Security note

To implement egress traffic control in a secure way, you must direct egress traffic through an egress gateway and address the security concerns expressed in Configure an Egress Gateway example, Additional Security Considerations.

Cleanup

Shutdown the sleep service:

Zip
$ kubectl delete -f @samples/sleep/sleep.yaml@