Record HTTP traffic

Tap filter configuration

0. Create a CR EnvoyHTTPFilter with a Tap filter configuration.
1.  

   Example

   apiVersion: microgateway.airlock.com/v1alpha1 
   kind: EnvoyHTTPFilter 
   metadata: 
     name: tap-filter 
   spec:  
     value:  
       name: envoy.filters.http.tap  
       typed_config:  
         "@type": type.googleapis.com/envoy.extensions.filters.http.tap.v3.Tap  
         common_config:  
           admin_config:  
             config_id: tap_filter_config
2. Reference the CR EnvoyHTTPFilter in the CR SidecarGateway.

3. Example SidecarGateway:

4.  

   Example

   apiVersion: microgateway.airlock.com/v1alpha1
   kind: SidecarGateway
   metadata:
     name: sidecar-gateway-example
   spec:
     ...
     applications:
       - containerPort: 8080
         ...
         envoyHTTPFilterRefs:
           prepend:
             - name: tap-filter
   
5. Configure a port forwarding for the Envoy Administration interface as described in Envoy Administration interface. The standard port of the interface is port 19000.
6.  

   Terminal box

   kubectl -n <APPLICATION_NAMESPACE> port-forward <POD_NAME> 19000:19000
7. Run the following CURL command to tap into the traffic.
8.  

   Terminal box

   curl localhost:19000/tap --data '{ 
     "config_id": "tap_filter_config", 
     "tap_config": { 
       "match_config": { 
         "any_match": "true" 
       }, 
       "output_config": { 
         "sinks": [ 
           { 
             "streaming_admin": {}, 
             "format": "JSON_BODY_AS_STRING" 
           }         
         ], 
         "max_buffered_rx_bytes": 100000
       } 
     } 
   }'
9.  

   Notice

   • The config_id value in the CURL command must match the config_id value configured in the CR EnvoyHTTPFilter.
   • With this config, all requests up to a size of 100000 bytes are recorded in JSON format.
   • The documented CURL command uses a configuration to record all requests. To record specific requests, create your match_config. For more information refer to official Envoy Tap filter documentation.
10. The tapped traffic arriving at the Microgateway is output to the console. Consider writing the output to a file, as the output can generate a considerable amount of data.

Tap filter configuration

0. Create a CR EnvoyHTTPFilter with a Tap filter configuration.
1.  

   Example

   apiVersion: microgateway.airlock.com/v1alpha1 
   kind: EnvoyHTTPFilter 
   metadata: 
     name: tap-filter 
   spec:  
     value:  
       name: envoy.filters.http.tap  
       typed_config:  
         "@type": type.googleapis.com/envoy.extensions.filters.http.tap.v3.Tap  
         common_config:  
           admin_config:  
             config_id: tap_filter_config
2. Create a CR EnvoyExtensionPolicy and refer to the desired HTTPRoute. Reference the CR EnvoyHTTPFilter.

3. Example EnvoyExtensionPolicy:

4.  

   Example

   apiVersion: microgateway.airlock.com/v1alpha1
   kind: EnvoyExtensionPolicy
   metadata:
     name: envoy-extension-policy-example
   spec:
     targetRefs:
       - group: gateway.networking.k8s.io
         kind: HTTPRoute
         name: echoserver
     envoyHTTPFilterRefs:
       prepend:
         - name: tap-filter
   
5.  

   Notice

   The Gateway of the targeted HTTPRoutes must be configured to allow EnvoyExtensionPolicy in its GatewayParameters or this Policy will have no effect.

   Set spec.features.envoyExtensionPolicyEnabled: true in the CR GatewayParameters of the Gateway of the targeted HTTPRoute.
6. Configure a port forwarding for the Envoy Administration interface as described in Envoy Administration interface. The standard port of the interface is port 19000.
7.  

   Terminal box

   kubectl -n <GATEWAY_NAMESPACE> port-forward <GATEWAY_POD_NAME> 19000:19000
8. Run the following CURL command to tap into the traffic.
9.  

   Terminal box

   curl localhost:19000/tap --data '{ 
     "config_id": "tap_filter_config", 
     "tap_config": { 
       "match_config": { 
         "any_match": "true" 
       }, 
       "output_config": { 
         "sinks": [ 
           { 
             "streaming_admin": {}, 
             "format": "JSON_BODY_AS_STRING" 
           }         
         ], 
         "max_buffered_rx_bytes": 100000
       } 
     } 
   }'
10.  

    Notice

    • The config_id value in the CURL command must match the config_id value configured in the CR EnvoyHTTPFilter.
    • With this config, all requests up to a size of 100000 bytes are recorded in JSON format.
    • The documented CURL command uses a configuration to record all requests. To record specific requests, create your match_config. For more information refer to official Envoy Tap filter documentation.
11. The tapped traffic arriving at the Microgateway is output to the console. Consider writing the output to a file, as the output can generate a considerable amount of data.

1. Run the CURL command and write the output to a file.

2. Example to save the tapped traffic in a file:

3.  

   Terminal box

   curl localhost:19000/tap --data '{ 
     ...
   }' > myfile.json
4. Analyze the JSON file with jq

5. Example to analyze with jq:

6.  

   Terminal box

   jq -c '
     .http_buffered_trace as $t |
     {
       method: ($t.request.headers[] | select(.key==":method") | .value),
       host:   ($t.request.headers[] | select(.key==":authority") | .value),
       path:   ($t.request.headers[] | select(.key==":path") | .value),
       status: ($t.response.headers[]|select(.key==":status")|.value|tonumber),
       req_bytes:  ($t.request.body.as_string|tostring|length),
       resp_bytes: ($t.response.body.as_string|tostring|length)
     }
   ' myfile.json
   

7. Example output

8.  

   Example

   {"method":"GET","host":"nextcloud-127-0-0-1.nip.io","path":"/status.php","status":200,"req_bytes":4,"resp_bytes":170}
   {"method":"PUT","host":"nextcloud-127-0-0-1.nip.io","path":"/ocs/v2.php/apps/user_status/api/v1/heartbeat?format=json","status":200,"req_bytes":19,"resp_bytes":223}
   {"method":"GET","host":"nextcloud-127-0-0-1.nip.io","path":"/apps/dashboard/","status":200,"req_bytes":4,"resp_bytes":1024}
   {"method":"GET","host":"nextcloud-127-0-0-1.nip.io","path":"/ocs/v2.php/apps/dashboard/api/v1/widgets","status":200,"req_bytes":4,"resp_bytes":1024}
   {"method":"PROPFIND","host":"nextcloud-127-0-0-1.nip.io","path":"/remote.php/dav/files/admin/","status":207,"req_bytes":566,"resp_bytes":1024}
   {"method":"GET","host":"nextcloud-127-0-0-1.nip.io","path":"/ocs/v2.php/apps/recommendations/api/v1/recommendations/always","status":200,"req_bytes":4,"resp_bytes":1024}