Summary

After upgrade envoy-1.29 to envoy-1.31.* started observing increased number of failures on downstream services, failing with parsing errors etc.

Details

setup where issue is reproduceable, however could not find a minimal working environment ALB->HTTP/1.1->EnvoyA->HTTP2->EnvoyB->HTTP/1.1->Custom HTTP/1.1 SRV

EnvoyB performs queueing with disabled outlier detection through upstream cluster config:

    outlier_detection:
      consecutive_5xx: 0
      enforcing_consecutive_5xx: 0
      consecutive_gateway_failure: 0
      enforcing_consecutive_gateway_failure: 0
      split_external_local_origin_errors: true
      consecutive_local_origin_failure: 0
      enforcing_consecutive_local_origin_failure: 0
      enforcing_local_origin_success_rate: 0
    circuit_breakers:
      thresholds:
      - priority: DEFAULT
        max_connections: 1536
        max_requests: 2048
        max_pending_requests: 2048
        max_retries: 512
      - priority: HIGH
        max_connections: 1536
        max_requests: 2048
        max_pending_requests: 2048
        max_retries: 512
      per_host_thresholds:
      - priority: DEFAULT
        max_connections: 3
      - priority: HIGH
        max_connections: 3
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        common_http_protocol_options:
          idle_timeout: 90s
          max_requests_per_connection: 64
        explicit_http_config:
          http_protocol_options:
            allow_chunked_length: false
            override_stream_error_on_invalid_http_message: false

Original hypothesis was that some of requests got entangled, which was confirmed during research. Investigation leads to envoy.reloadable_features.http1_balsa_delay_reset flag which is set to true by default, together with another leading observation that HEAD request could not flow though envoyB always failing through UT/DR (depends on client timeout settings) and parser failure when SRV response is coupled with 102 Processing here is the log of simple curl request:

the theory is that configuration somehow triggers the problem in the parser and make it even worse if response from the server is not
well formatted single response, as a result other components become unreliable, while we would UPE or something similar.

PoC

in test environment were able to easily reproduce by submitting HEAD requests, and server response as sequence of:

HTTP/1.1 102 Processing\r\n
\r\n
HTTP/1.1 200 Ok \r\n
......

25 clients. one instance of envoyB, all requests hitting same SRV instance(process is sequential), Client timeout ~ 150ms, Request Duration ~30ms, Avg Response on client ~ 440ms,, 65% of request failing with DC on envoyA, %50 of requests got to envoyB failing with DR, rest of requests has 200 ok

was not able to reproduce the request entanglement without keep-alive max_requests_per_connection: 1 or with disabled delayed reset: envoy.reloadable_features.http1_balsa_delay_reset=false

Impact

High, users who are using keep-alive http/1.1 upstream and/or vulnerable upstream server which not strictly posing single well formatted responses

Mitigation

• Disable envoy.reloadable_features.http1_balsa_delay_reset