fix: close idle OTLP connections before the collector does
otelsetup / test (push) Skipped
otelsetup / vulnerabilities (push) Skipped
pre-commit / pre-commit (push) Skipped
otelsetup / vulnerabilities (pull_request) Successful in 1m0s
otelsetup / test (pull_request) Successful in 1m21s
pre-commit / pre-commit (pull_request) Successful in 3m36s

Metrics are pushed every 60s and Alloy's OTLP receiver closes connections idle for 1m, while the exporters keep them for 90s. A push could reuse a connection the collector was closing and fail with EOF or connection reset; the data was dropped (no retry for a POST or for transport errors). In prod that was ~3 failed metric pushes per pod per hour, plus occasional trace batches.

Both exporters now use an HTTP client that closes idle connections after 30s. A custom client makes the exporters ignore the OTLP timeout and certificate env vars; none are set.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rfQ5HJ7zwfuYQcWXd3Mm3
This commit is contained in:
argoyleandClaude Opus 5 committed 2026-09-19 15:43:14 +02:00
1 parent 80d749ab92
commit f1551a5f47
3 files changed
+56 -2

No files matched your search

+1
View File
@@ -42,3 +42,4 @@ http.Handle("/", otelsetup.Handler(myHandler))
- `OTEL_RESOURCE_ATTRIBUTES` - Auto-set if not provided (service.name, service.version, service.environment)
- Standard OTLP environment variables for endpoint configuration
- The trace and metric exporters use our own HTTP client (idle connections closed after 30s, before the collector's 1m idle timeout), so `OTEL_EXPORTER_OTLP_*TIMEOUT` and the `OTEL_EXPORTER_OTLP_*CERTIFICATE` / `*CLIENT_*` TLS variables are ignored