Lightstep

This task shows you how to configure Istio to collect trace spans and send them to Lightstep. Lightstep lets you analyze 100% of unsampled transaction data from large scale production software to produce meaningful distributed traces and metrics that help explain performance behaviors and accelerate root cause analysis. At the end of this task, Istio sends trace spans from the proxies to a Lightstep Satellite pool making them available to the web UI. By default, all HTTP requests are captured (to see end-to-end traces, your code needs to forward OT headers even if it does not join the traces).

If you only want to collect tracing spans directly from Istio (and not add specific instrumentation directly to your code), then you don’t need to configure any tracers, as long as your services forward the HTTP headers generated by traces.

This task uses the Bookinfo sample application as an example.

Before you begin

1. Ensure you have a Lightstep account. Sign up for a free trial of Lightstep.

2. If you’re using on-premise Satellites, ensure you have a satellite pool configured with TLS certs and a secure GRPC port exposed. See Install and Configure Satellites for details about setting up satellites.

   For Lightstep Public Satellites or Developer Satellites, your satellites are already configured. However you need to download this certificate to a local directory.

3. Ensure sure you have a Lightstep access token. Access tokens allow your app to communicate with your Lightstep project.

Deploy Istio

How you deploy Istio depends on which type of Satellite you use.

Deploy Istio with On-Premise Satellites

These instructions do not assume TLS. If you are using TLS for your Satellite pool, follow the config for the Public Satellite pool, but use your own cert and your own pool’s endpoint (host:port).

1. You need to deploy Istio with your Satellite address at an address in the format <Host>:<Port>, for example lightstep-satellite.lightstep:9292. You find this in your configuration file.

2. Deploy Istio with the following configuration parameters specified:

   • global.proxy.tracer="lightstep"
   • meshConfig.defaultConfig.tracing.sampling=100
   • meshConfig.defaultConfig.tracing.lightstep.address="<satellite-address>"
   • meshConfig.defaultConfig.tracing.lightstep.accessToken="<access-token>"

   You can set these parameters using the --set key=value syntax when you run the install command. For example:

   $ istioctl install \
       --set global.proxy.tracer="lightstep" \
       --set meshConfig.defaultConfig.tracing.sampling=100 \
       --set meshConfig.defaultConfig.tracing.lightstep.address="<satellite-address>" \
       --set meshConfig.defaultConfig.tracing.lightstep.accessToken="<access-token>" \
   

Deploy Istio with Public or Developer Mode Satellites

Follow these steps if you’re using the Public or Developer Mode Satellites, or if you’re using on-premise Satellites with a TLS certificate.

1. Store your satellite pool’s certificate authority certificate as a secret in the default and istio-system namespace, the latter for use by the Istio gateways. Download and use this certificate. If you deploy the Bookinfo application in a different namespace, create the secret in that namespace instead.

   $ CACERT=$(cat Cert_Auth.crt | base64) # Cert_Auth.crt contains the necessary CACert
   $ NAMESPACE=default
   

   $ cat <<EOF | kubectl apply -f -
     apiVersion: v1
     kind: Secret
     metadata:
       name: lightstep.cacert
       namespace: $NAMESPACE
       labels:
         app: lightstep
     type: Opaque
     data:
       cacert.pem: $CACERT
   EOF
   

2. Deploy Istio with the following configuration parameters specified:

   global:
     proxy:
       tracer: "lightstep"
   meshConfig:
     defaultConfig:
       tracing:
         lightstep:
           address: "ingest.lightstep.com:443"
           accessToken: "<access-token>"
         sampling: 100
         tlsSettings
           mode: "SIMPLE"
           # Specifying ca certificate here will mount `lightstep.cacert` secret volume
           # at all sidecars by default.
           caCertificates="/etc/lightstep/cacert.pem"
   components:
     ingressGateways:
     # `lightstep.cacert` secret volume needs to be mount at gateways via k8s overlay.
     - name: istio-ingressgateway
       enabled: true
       k8s:
         overlays:
         - kind: Deployment
           name: istio-ingressgateway
           patches:
           - path: spec.template.spec.containers[0].volumeMounts[-1]
             value: |
               name: lightstep-certs
               mountPath: /etc/lightstep
               readOnly: true
           - path: spec.template.spec.volumes[-1]
             value: |
               name: lightstep-certs
               secret:
                 secretName: lightstep.cacert
                 optional: true
   

Install and run the Bookinfo app

1. Follow the instructions to deploy the Bookinfo sample application.

2. Follow the instructions to create an ingress gateway for the Bookinfo application.

3. To verify the previous step’s success, confirm that you set GATEWAY_URL environment variable in your shell.

4. Send traffic to the sample application.

   $ curl http://$GATEWAY_URL/productpage
   

Visualize trace data

1. Load the Lightstep web UI. You’ll see the three Bookinfo services listed in the Service Directory.

   

2. Navigate to the Explorer view.

   

3. Find the query bar at the top. The query bar allows you to interactively filter results by a Service, Operation, and Tag values.

4. Select productpage.default from the Service drop-down list.

5. Click Run. You see something similar to the following:

   

6. Click on the first row in the table of example traces below the latency histogram to see the details corresponding to your refresh of the /productpage. The page then looks similar to:

   

The screenshot shows that the trace is comprised of a set of spans. Each span corresponds to a Bookinfo service invoked during the execution of a /productpage request.

Two spans in the trace represent every RPC. For example, the call from productpage to reviews starts with the span labeled with the reviews.default.svc.cluster.local:9080/* operation and the productpage.default: proxy client service. This service represents the client-side span of the call. The screenshot shows that the call took 15.30 ms. The second span is labeled with the reviews.default.svc.cluster.local:9080/* operation and the reviews.default: proxy server service. The second span is a child of the first span and represents the server-side span of the call. The screenshot shows that the call took 14.60 ms.

Trace sampling

Istio captures traces at a configurable trace sampling percentage. To learn how to modify the trace sampling percentage, visit the Distributed Tracing trace sampling section.

When using Lightstep, we do not recommend reducing the trace sampling percentage below 100%. To handle a high traffic mesh, consider scaling up the size of your satellite pool.

Cleanup

If you are not planning any follow-up tasks, remove the Bookinfo sample application and any Lightstep secrets from your cluster.

1. To remove the Bookinfo application, refer to the Bookinfo cleanup instructions.

2. Remove the secret generated for Lightstep:

$ kubectl delete secret lightstep.cacert