Summary

Envoy exposed an out-of-memory (OOM) vector from the mirror response, since async HTTP client will buffer the response with an unbounded buffer.

Details

If a mirror cluster decides to send extremely large response messages, those messages are fully buffered in memory by the async HTTP client. This allows a malicious backend to potentially OOM Envoys by sending huge responses.

Other components that are using HTTP async client in Envoy may also be impacted.

• wasm filter
• lua filter
• ext_proc
• oauth fitler
• ext_authz
• jwks_fetcher
• gcp_auther_filter
• aws_metadata_fetcher
• opentelemetry/http_trace_exporter,
• opentelemetry/dynatrace/sampler_config_provider
• config_subscription/rest/rest_api_fetcher
• rate_limiter

Most of the auth/log related extensions assume trusted upstream, while wasm, lua, and ext_proc may not assume it.

Impact

Denial of service and Envoy will be OOMed.

Mitigation

Patched versions:

• disable buffering the mirror response, as Envoy will never use it.
• provide a configuration for the HTTP async client with a default hard limit. Users can set the limit through a runtime key based on needs, protection mechanism, security posture, etc.

Credit

Reported by: Paul Ogilby [email protected] (@paul-r-gall)