Envoy Statistics

The Envoy proxy keeps detailed statistics about network traffic.

Envoy’s statistics only cover the traffic for a particular Envoy instance. See Observability for persistent per-service Istio telemetry. The statistics the Envoy proxies record can provide more information about specific pod instances.

To see the statistics for a pod:

$ kubectl exec "$POD" -c istio-proxy -- pilot-agent request GET stats

Envoy generates statistics about its behavior, scoping the statistics by proxy function. Examples include:

• Upstream connection
• Listener
• HTTP Connection Manager
• TCP proxy
• Router

By default, Istio configures Envoy to record a minimal set of statistics to reduce the overall CPU and memory footprint of the installed proxies. The default collection keys are:

• cluster_manager
• listener_manager
• server
• cluster.xds-grpc

To see the Envoy settings for statistics data collection use istioctl proxy-config bootstrap and follow the deep dive into Envoy configuration. Envoy only collects statistical data on items matching the inclusion_list within the stats_matcher JSON element.

To configure Istio proxy to record additional statistics, you can add ProxyConfig.ProxyStatsMatcher to your mesh config. For example, to enable stats for circuit breakers, request retries, upstream connections, and request timeouts globally, you can specify stats matcher as follows:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    defaultConfig:
      proxyStatsMatcher:
        inclusionRegexps:
          - ".*outlier_detection.*"
          - ".*upstream_rq_retry.*"
          - ".*upstream_cx_.*"
        inclusionSuffixes:
          - "upstream_rq_timeout"

You can also override the global stats matching configuration per proxy by using the proxy.istio.io/config annotation. For example, to configure the same stats generation inclusion as above, you can add the annotation to a gateway proxy or a workload as follows:

metadata:
  annotations:
    proxy.istio.io/config: |-
      proxyStatsMatcher:
        inclusionRegexps:
        - ".*outlier_detection.*"
        - ".*upstream_rq_retry.*"
        - ".*upstream_cx_.*"
        inclusionSuffixes:
        - "upstream_rq_timeout"

Istio naming of Envoy resources

Envoy scopes most statistics by the cluster, connection manager, or listener address they belong to. Istio generates the cluster and connection manager names when it translates the mesh configuration into Envoy configuration, so they also appear in statistic names, Prometheus labels, and access logs.

These names apply to sidecar proxies and gateways. Waypoint proxies in ambient mode use different cluster names, and ztunnel does not use Envoy.

The subset segment of an outbound cluster name is empty when no DestinationRule subset applies, for example outbound|9080||reviews.default.svc.cluster.local. Envoy creates the traffic statistics of a cluster, such as upstream_rq_total, when the proxy first sends traffic to it, so they are missing for clusters that have not received traffic yet. To list the clusters and listeners of a proxy, see debugging Envoy and Istiod.

Istio ends most cluster and HTTP connection manager names used in statistics with a ; delimiter, for example:

cluster.outbound|9080||reviews.default.svc.cluster.local;.upstream_rq_total
http.outbound_0.0.0.0_9080;.downstream_rq_total

The proxy bootstrap configuration extracts these names into tags, which become labels in the Prometheus output of Envoy. For example, envoy_cluster_upstream_rq_total has a cluster_name label with the value outbound|9080||reviews.default.svc.cluster.local. The most commonly used tags are:

• cluster_name: the cluster name.
• http_conn_manager_prefix: the HTTP connection manager name.
• tcp_prefix: the TCP proxy name.
• response_code and response_code_class: the HTTP response code, such as 200 or 2xx.
• listener_address: the address of the listener, such as 0.0.0.0_15006.

To see the full list of tags and the expressions that extract them, look at the stats_config element in the output of istioctl proxy-config bootstrap.

Customize cluster stat names

The inboundClusterStatName and outboundClusterStatName mesh config options replace the cluster names in statistics with a pattern that you control. The names of the clusters themselves do not change, so routing and istioctl proxy-config output are not affected. The pattern can use the following variables:

Istio appends the ; delimiter to the result, so the cluster_name tag keeps working. The same patterns also name the statistics of TCP proxies, without the delimiter. They do not change the cluster names in access logs.

If a pattern leaves out %SUBSET_NAME%, the clusters of all subsets of a service share the same statistics. For example, to name the outbound cluster statistics after the service and port:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    outboundClusterStatName: "%SERVICE%_%SERVICE_PORT%"

With this setting, the statistics for requests to the httpbin service on port 8000 in the default namespace start with cluster.httpbin.default_8000; instead of cluster.outbound|8000||httpbin.default.svc.cluster.local;.