Consuming External Web Services

Mesh-external service entries for egress HTTPS traffic

By Vadim Eisenberg | January 31, 2018 (updated on April 11, 2019) | 9 minute read

In many cases, not all the parts of a microservices-based application reside in a service mesh. Sometimes, the microservices-based applications use functionality provided by legacy systems that reside outside the mesh. You may want to migrate these systems to the service mesh gradually. Until these systems are migrated, they must be accessed by the applications inside the mesh. In other cases, the applications use web services provided by third parties.

In this blog post, I modify the Istio Bookinfo Sample Application to fetch book details from an external web service (Google Books APIs). I show how to enable egress HTTPS traffic in Istio by using mesh-external service entries. I provide two options for egress HTTPS traffic and describe the pros and cons of each of the options.

Initial setting

To demonstrate the scenario of consuming an external web service, I start with a Kubernetes cluster with Istio installed. Then I deploy Istio Bookinfo Sample Application. This application uses the details microservice to fetch book details, such as the number of pages and the publisher. The original details microservice provides the book details without consulting any external service.

The example commands in this blog post work with Istio 1.0+, with or without mutual TLS enabled. The Bookinfo configuration files reside in the samples/bookinfo directory of the Istio release archive.

Here is a copy of the end-to-end architecture of the application from the original Bookinfo sample application.

Perform the steps in the Deploying the application, Confirm the app is running, Apply default destination rules sections, and change Istio to the blocking-egress-by-default policy.

Bookinfo with HTTPS access to a Google Books web service

Deploy a new version of the details microservice, v2, that fetches the book details from Google Books APIs. Run the following command; it sets the DO_NOT_ENCRYPT environment variable of the service’s container to false. This setting will instruct the deployed service to use HTTPS (instead of HTTP) to access to the external service.

$ kubectl apply -f samples/bookinfo/platform/kube/bookinfo-details-v2.yaml⁴ --dry-run -o yaml | kubectl set env --local -f - 'DO_NOT_ENCRYPT=false' -o yaml | kubectl apply -f -

The updated architecture of the application now looks as follows:

The Bookinfo Application with details V2

Note that the Google Books web service is outside the Istio service mesh, the boundary of which is marked by a dashed line.

Now direct all the traffic destined to the details microservice, to details version v2.

$ kubectl apply -f samples/bookinfo/networking/virtual-service-details-v2.yaml⁵

Note that the virtual service relies on a destination rule that you created in the Apply default destination rules section.

Access the web page of the application, after determining the ingress IP and port.

Oops… Instead of the book details you have the Error fetching product details message displayed:

The good news is that your application did not crash. With a good microservice design, you do not have failure propagation. In your case, the failing details microservice does not cause the productpage microservice to fail. Most of the functionality of the application is still provided, despite the failure in the details microservice. You have graceful service degradation: as you can see, the reviews and the ratings are displayed correctly, and the application is still useful.

So what might have gone wrong? Ah… The answer is that I forgot to tell you to enable traffic from inside the mesh to an external service, in this case to the Google Books web service. By default, the Istio sidecar proxies (Envoy proxies) block all the traffic to destinations outside the cluster. To enable such traffic, you must define a mesh-external service entry.

Enable HTTPS access to a Google Books web service

No worries, define a mesh-external service entry and fix your application. You must also define a virtual service to perform routing by SNI to the external service.

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: googleapis
spec:
  hosts:
  - www.googleapis.com
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  location: MESH_EXTERNAL
  resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: googleapis
spec:
  hosts:
  - www.googleapis.com
  tls:
  - match:
    - port: 443
      sni_hosts:
      - www.googleapis.com
    route:
    - destination:
        host: www.googleapis.com
        port:
          number: 443
      weight: 100
EOF

Now accessing the web page of the application displays the book details without error:

You can query your service entries:

$ kubectl get serviceentries
NAME         AGE
googleapis   8m

You can delete your service entry:

$ kubectl delete serviceentry googleapis
serviceentry "googleapis" deleted

and see in the output that the service entry is deleted.

Accessing the web page after deleting the service entry produces the same error that you experienced before, namely Error fetching product details. As you can see, the service entries are defined dynamically, as are many other Istio configuration artifacts. The Istio operators can decide dynamically which domains they allow the microservices to access. They can enable and disable traffic to the external domains on the fly, without redeploying the microservices.

Cleanup of HTTPS access to a Google Books web service

$ kubectl delete serviceentry googleapis
$ kubectl delete virtualservice googleapis
$ kubectl delete -f samples/bookinfo/networking/virtual-service-details-v2.yaml⁵
$ kubectl delete -f samples/bookinfo/platform/kube/bookinfo-details-v2.yaml⁴

TLS origination by Istio

There is a caveat to this story. Suppose you want to monitor which specific set of Google APIs your microservices use (Books, Calendar, Tasks etc.) Suppose you want to enforce a policy that using only Books APIs is allowed. Suppose you want to monitor the book identifiers that your microservices access. For these monitoring and policy tasks you need to know the URL path. Consider for example the URL www.googleapis.com/books/v1/volumes?q=isbn:0486424618. In that URL, Books APIs is specified by the path segment /books, and the ISBN number by the path segment /volumes?q=isbn:0486424618. However, in HTTPS, all the HTTP details (hostname, path, headers etc.) are encrypted and such monitoring and policy enforcement by the sidecar proxies is not possible. Istio can only know the server name of the encrypted requests by the SNI (Server Name Indication) field, in this case www.googleapis.com.

To allow Istio to perform monitoring and policy enforcement of egress requests based on HTTP details, the microservices must issue HTTP requests. Istio then opens an HTTPS connection to the destination (performs TLS origination). The code of the microservices must be written differently or configured differently, according to whether the microservice runs inside or outside an Istio service mesh. This contradicts the Istio design goal of maximizing transparency. Sometimes you need to compromise…

The diagram below shows two options for sending HTTPS traffic to external services. On the top, a microservice sends regular HTTPS requests, encrypted end-to-end. On the bottom, the same microservice sends unencrypted HTTP requests inside a pod, which are intercepted by the sidecar Envoy proxy. The sidecar proxy performs TLS origination, so the traffic between the pod and the external service is encrypted.

HTTPS traffic to external services, with TLS originated by the microservice vs. by the sidecar proxy

Here is how both patterns are supported in the Bookinfo details microservice code, using the Ruby net/http module:

uri = URI.parse('https://www.googleapis.com/books/v1/volumes?q=isbn:' + isbn)
http = Net::HTTP.new(uri.host, ENV['DO_NOT_ENCRYPT'] === 'true' ? 80:443)
...
unless ENV['DO_NOT_ENCRYPT'] === 'true' then
     http.use_ssl = true
end

When the DO_NOT_ENCRYPT environment variable is defined, the request is performed without SSL (plain HTTP) to port 80.

You can set the DO_NOT_ENCRYPT environment variable to “true” in the Kubernetes deployment spec of details v2, the container section:

env:
- name: DO_NOT_ENCRYPT
  value: "true"

In the next section you will configure TLS origination for accessing an external web service.

Bookinfo with TLS origination to a Google Books web service

Deploy a version of details v2 that sends an HTTP request to Google Books APIs. The DO_NOT_ENCRYPT variable is set to true in bookinfo-details-v2.yaml.
Zip
```
$ kubectl apply -f samples/bookinfo/platform/kube/bookinfo-details-v2.yaml⁴
```

Direct the traffic destined to the details microservice, to details version v2.

$ kubectl apply -f samples/bookinfo/networking/virtual-service-details-v2.yaml⁵

Create a mesh-external service entry for www.google.apis , a virtual service to rewrite the destination port from 80 to 443, and a destination rule to perform TLS origination.

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: googleapis
spec:
  hosts:
  - www.googleapis.com
  ports:
  - number: 80
    name: http
    protocol: HTTP
  - number: 443
    name: https
    protocol: HTTPS
  resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: rewrite-port-for-googleapis
spec:
  hosts:
  - www.googleapis.com
  http:
  - match:
    - port: 80
    route:
    - destination:
        host: www.googleapis.com
        port:
          number: 443
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: originate-tls-for-googleapis
spec:
  host: www.googleapis.com
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 443
      tls:
        mode: SIMPLE # initiates HTTPS when accessing www.googleapis.com
EOF

Access the web page of the application and verify that the book details are displayed without errors.
Enable Envoy’s access logging

Check the log of of the sidecar proxy of details v2 and see the HTTP request.

$ kubectl logs $(kubectl get pods -l app=details -l version=v2 -o jsonpath='{.items[0].metadata.name}') istio-proxy | grep googleapis
[2018-08-09T11:32:58.171Z] "GET /books/v1/volumes?q=isbn:0486424618 HTTP/1.1" 200 - 0 1050 264 264 "-" "Ruby" "b993bae7-4288-9241-81a5-4cde93b2e3a6" "www.googleapis.com:80" "172.217.20.74:80"
EOF

Note the URL path in the log, the path can be monitored and access policies can be applied based on it. To read more about monitoring and access policies for HTTP egress traffic, check out this blog post.

Cleanup of TLS origination to a Google Books web service

$ kubectl delete serviceentry googleapis
$ kubectl delete virtualservice rewrite-port-for-googleapis
$ kubectl delete destinationrule originate-tls-for-googleapis
$ kubectl delete -f samples/bookinfo/networking/virtual-service-details-v2.yaml⁵
$ kubectl delete -f samples/bookinfo/platform/kube/bookinfo-details-v2.yaml⁴

Relation to Istio mutual TLS

Note that the TLS origination in this case is unrelated to the mutual TLS applied by Istio. The TLS origination for the external services will work, whether the Istio mutual TLS is enabled or not. The mutual TLS secures service-to-service communication inside the service mesh and provides each service with a strong identity. The external services in this blog post were accessed using one-way TLS, the same mechanism used to secure communication between a web browser and a web server. TLS is applied to the communication with external services to verify the identity of the external server and to encrypt the traffic.

Conclusion

In this blog post I demonstrated how microservices in an Istio service mesh can consume external web services by HTTPS. By default, Istio blocks all the traffic to the hosts outside the cluster. To enable such traffic, mesh-external service entries must be created for the service mesh. It is possible to access the external sites either by issuing HTTPS requests, or by issuing HTTP requests with Istio performing TLS origination. When the microservices issue HTTPS requests, the traffic is encrypted end-to-end, however Istio cannot monitor HTTP details like the URL paths of the requests. When the microservices issue HTTP requests, Istio can monitor the HTTP details of the requests and enforce HTTP-based access policies. However, in that case the traffic between microservice and the sidecar proxy is unencrypted. Having part of the traffic unencrypted can be forbidden in organizations with very strict security requirements.