Record HTTP Traffic

The Envoy Tap filter allows to interpose on and record HTTP traffic that the Microgateway receives. For further details, see official Envoy Tap filter documentation.

Setup the Tap filter

Tap filter configuration

  1. Create a CR EnvoyHTTPFilter with a Tap filter configuration.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

    apiVersion: microgateway.airlock.com/v1alpha1 
kind: EnvoyHTTPFilter 
metadata: 
  name: tap-filter 
spec:  
  value:  
    name: envoy.filters.http.tap  
    typed_config:  
      "@type": type.googleapis.com/envoy.extensions.filters.http.tap.v3.Tap  
      common_config:  
        admin_config:  
          config_id: tap_filter_config

  2. Create a CR EnvoyExtensionPolicy and refer to the desired HTTPRoute. Reference the CR EnvoyHTTPFilter.

    Example EnvoyExtensionPolicy:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

    apiVersion: microgateway.airlock.com/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: envoy-extension-policy-example
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: echoserver
  envoyHTTPFilterRefs:
    prepend:
      - name: tap-filter
    Notice

    The Gateway of the targeted HTTPRoutes must be configured to allow EnvoyExtensionPolicy in its GatewayParameters or this Policy will have no effect.

    • Set spec.features.envoyExtensionPolicyEnabled: true in the CR GatewayParameters of the Gateway of the targeted HTTPRoute.

  3. Configure a port forwarding for the Envoy Administration interface as described in Envoy Administration Interface. The standard port of the interface is port 19000.

    1

    kubectl -n <GATEWAY_NAMESPACE> port-forward <GATEWAY_POD_NAME> 19000:19000

  4. Run the following CURL command to tap into the traffic.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

    curl localhost:19000/tap --data '{ 
  "config_id": "tap_filter_config", 
  "tap_config": { 
    "match_config": { 
      "any_match": "true" 
    }, 
    "output_config": { 
      "sinks": [ 
        { 
          "streaming_admin": {}, 
          "format": "JSON_BODY_AS_STRING" 
        }         
      ], 
      "max_buffered_rx_bytes": 100000
    } 
  } 
}'
    Notice
    • The config_id value in the CURL command must match the config_id value configured in the CR EnvoyHTTPFilter.
    • With this config, all requests up to a size of 100000 bytes are recorded in JSON format.
    • The documented CURL command uses a configuration to record all requests. To record specific requests, create your match_config. For more information refer to official Envoy Tap filter documentation.

    The tapped traffic arriving at the Microgateway is output to the console. Consider writing the output to a file, as the output can generate a considerable amount of data.

Analyze the tapped traffic

  1. Run the CURL command and write the output to a file.

    Example to save the tapped traffic in a file:

    1
2
3

    curl localhost:19000/tap --data '{ 
  ...
}' > myfile.json

  2. Analyze the JSON file with jq

    Example to analyze with jq:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

    jq -c '
  .http_buffered_trace as $t |
  {
    method: ($t.request.headers[] | select(.key==":method") | .value),
    host:   ($t.request.headers[] | select(.key==":authority") | .value),
    path:   ($t.request.headers[] | select(.key==":path") | .value),
    status: ($t.response.headers[]|select(.key==":status")|.value|tonumber),
    req_bytes:  ($t.request.body.as_string|tostring|length),
    resp_bytes: ($t.response.body.as_string|tostring|length)
  }
' myfile.json

    Example output

    1
2
3
4
5
6

    {"method":"GET","host":"nextcloud-127-0-0-1.nip.io","path":"/status.php","status":200,"req_bytes":4,"resp_bytes":170}
{"method":"PUT","host":"nextcloud-127-0-0-1.nip.io","path":"/ocs/v2.php/apps/user_status/api/v1/heartbeat?format=json","status":200,"req_bytes":19,"resp_bytes":223}
{"method":"GET","host":"nextcloud-127-0-0-1.nip.io","path":"/apps/dashboard/","status":200,"req_bytes":4,"resp_bytes":1024}
{"method":"GET","host":"nextcloud-127-0-0-1.nip.io","path":"/ocs/v2.php/apps/dashboard/api/v1/widgets","status":200,"req_bytes":4,"resp_bytes":1024}
{"method":"PROPFIND","host":"nextcloud-127-0-0-1.nip.io","path":"/remote.php/dav/files/admin/","status":207,"req_bytes":566,"resp_bytes":1024}
{"method":"GET","host":"nextcloud-127-0-0-1.nip.io","path":"/ocs/v2.php/apps/recommendations/api/v1/recommendations/always","status":200,"req_bytes":4,"resp_bytes":1024}

External links: