Record HTTP traffic

Tap filter configuration

0. Create a CR EnvoyHTTPFilter with a Tap filter configuration.
1.  

   Example

   apiVersion: microgateway.airlock.com/v1alpha1 
   kind: EnvoyHTTPFilter 
   metadata: 
     name: tap-filter 
   spec:  
     value:  
       name: envoy.filters.http.tap  
       typed_config:  
         "@type": type.googleapis.com/envoy.extensions.filters.http.tap.v3.Tap  
         common_config:  
           admin_config:  
             config_id: tap_filter_config
2. Reference the CR EnvoyHTTPFilter in the CR SidecarGateway.

3. Example SidecarGateway:

4.  

   Example

   apiVersion: microgateway.airlock.com/v1alpha1
   kind: SidecarGateway
   metadata:
     name: sidecar-gateway-example
   spec:
     ...
     applications:
       - containerPort: 8080
         ...
         envoyHTTPFilterRefs:
           prepend:
             - name: tap-filter
   
5. Configure a port forwarding for the Envoy Administration interface as described in Inspecting the Microgateway Engine via the Envoy administration interface. The standard port of the interface is port 19000.
6.  

   Terminal box

   kubectl -n <APPLICATION_NAMESPACE> port-forward <POD_NAME> 19000:19000
7. Run the following CURL command to tap into the traffic.
8.  

   Terminal box

   curl localhost:19000/tap --data '{ 
     "config_id": "tap_filter_config", 
     "tap_config": { 
       "match_config": { 
         "any_match": "true" 
       }, 
       "output_config": { 
         "sinks": [ 
           { 
             "streaming_admin": {}, 
             "format": "JSON_BODY_AS_STRING" 
           }         
         ], 
         "max_buffered_rx_bytes": 100000
       } 
     } 
   }'
9.  

   Notice

   • The config_id value in the CURL command must match the config_id value configured in the CR EnvoyHTTPFilter.
   • With this config, all requests up to a size of 100000 bytes are recorded in JSON format.
   • The documented CURL command uses a configuration to record all requests. To record specific requests, create your match_config. For more information refer to official Envoy Tap filter documentation.
10. The tapped traffic arriving at the Microgateway is output to the console. Consider writing the output to a file, as the output can generate a considerable amount of data.

Tap filter configuration

0. Create a CR EnvoyHTTPFilter with a Tap filter configuration.
1.  

   Example

   apiVersion: microgateway.airlock.com/v1alpha1 
   kind: EnvoyHTTPFilter 
   metadata: 
     name: tap-filter 
   spec:  
     value:  
       name: envoy.filters.http.tap  
       typed_config:  
         "@type": type.googleapis.com/envoy.extensions.filters.http.tap.v3.Tap  
         common_config:  
           admin_config:  
             config_id: tap_filter_config
2. Create a CR EnvoyExtensionPolicy and refer to the desired HTTPRoute. Reference the CR EnvoyHTTPFilter.

3. Example EnvoyExtensionPolicy:

4.  

   Example

   apiVersion: microgateway.airlock.com/v1alpha1
   kind: EnvoyExtensionPolicy
   metadata:
     name: envoy-extension-policy-example
   spec:
     targetRefs:
       - group: gateway.networking.k8s.io
         kind: HTTPRoute
         name: echoserver
     envoyHTTPFilterRefs:
       prepend:
         - name: tap-filter
   
5.  

   Notice

   The Gateway of the targeted HTTPRoutes must be configured to allow EnvoyExtensionPolicy in its GatewayParameters or this Policy will have no effect.

   Set spec.features.envoyExtensionPolicyEnabled: true in the CR GatewayParameters of the Gateway of the targeted HTTPRoute.
6. Configure a port forwarding for the Envoy Administration interface as described in Inspecting the Microgateway Engine via the Envoy administration interface. The standard port of the interface is port 19000.
7.  

   Terminal box

   kubectl -n <GATEWAY_NAMESPACE> port-forward <GATEWAY_POD_NAME> 19000:19000
8. Run the following CURL command to tap into the traffic.
9.  

   Terminal box

   curl localhost:19000/tap --data '{ 
     "config_id": "tap_filter_config", 
     "tap_config": { 
       "match_config": { 
         "any_match": "true" 
       }, 
       "output_config": { 
         "sinks": [ 
           { 
             "streaming_admin": {}, 
             "format": "JSON_BODY_AS_STRING" 
           }         
         ], 
         "max_buffered_rx_bytes": 100000
       } 
     } 
   }'
10.  

    Notice

    • The config_id value in the CURL command must match the config_id value configured in the CR EnvoyHTTPFilter.
    • With this config, all requests up to a size of 100000 bytes are recorded in JSON format.
    • The documented CURL command uses a configuration to record all requests. To record specific requests, create your match_config. For more information refer to official Envoy Tap filter documentation.
11. The tapped traffic arriving at the Microgateway is output to the console. Consider writing the output to a file, as the output can generate a considerable amount of data.

1. Run the CURL command and write the output to a file.

2. Example to save the tapped traffic in a file:

3.  

   Terminal box

   curl localhost:19000/tap --data '{ 
     ...
   }' > myfile.json
4. Analyze the JSON file with jq

5. Example to analyze with jq:

6.  

   Terminal box

   jq -c '
     .http_buffered_trace as $t |
     {
       method: ($t.request.headers[] | select(.key==":method") | .value),
       host:   ($t.request.headers[] | select(.key==":authority") | .value),
       path:   ($t.request.headers[] | select(.key==":path") | .value),
       status: ($t.response.headers[]|select(.key==":status")|.value|tonumber),
       req_bytes:  ($t.request.body.as_string|tostring|length),
       resp_bytes: ($t.response.body.as_string|tostring|length)
     }
   ' myfile.json
   

7. Example output

8.  

   Example

   {"method":"GET","host":"nextcloud-127-0-0-1.nip.io","path":"/status.php","status":200,"req_bytes":4,"resp_bytes":170}
   {"method":"PUT","host":"nextcloud-127-0-0-1.nip.io","path":"/ocs/v2.php/apps/user_status/api/v1/heartbeat?format=json","status":200,"req_bytes":19,"resp_bytes":223}
   {"method":"GET","host":"nextcloud-127-0-0-1.nip.io","path":"/apps/dashboard/","status":200,"req_bytes":4,"resp_bytes":1024}
   {"method":"GET","host":"nextcloud-127-0-0-1.nip.io","path":"/ocs/v2.php/apps/dashboard/api/v1/widgets","status":200,"req_bytes":4,"resp_bytes":1024}
   {"method":"PROPFIND","host":"nextcloud-127-0-0-1.nip.io","path":"/remote.php/dav/files/admin/","status":207,"req_bytes":566,"resp_bytes":1024}
   {"method":"GET","host":"nextcloud-127-0-0-1.nip.io","path":"/ocs/v2.php/apps/recommendations/api/v1/recommendations/always","status":200,"req_bytes":4,"resp_bytes":1024}