Understanding and Managing Late-Arriving Spans in Refinery

Late-arriving spans are spans that reach Refinery after the sampling decision for their trace has already been made. When that decision was to keep the trace, every late span is sent to Honeycomb at the trace's sample rate without counting toward your throughput goals, which can significantly impact your data volume and quota usage. When the decision was to drop the trace, late spans are dropped along with it.

What Are Late-Arriving Spans?

Refinery makes a sampling decision for a trace at one of these points:

• When the root span arrives, the trace is sent after the SendDelay period (default 2 seconds)

• If no root span arrives, the trace is sent when TraceTimeout expires (default 60 seconds)

• If SpanLimit is configured and a trace exceeds that span count, the decision is made immediately

Any span for that trace arriving after this point is a late-arriving span. Common causes include:

• Network delays or connectivity issues

• Heavy application load causing processing delays

• Long-running batch jobs or processes

• Infrastructure instability (like database outages)

• Very large traces that hit SpanLimit and are decided early, while the rest of the trace is still arriving

Why Late Spans Bypass Throughput Controls

Throughput-based samplers calculate against the span count known at decision time. If a trace is kept, all subsequent spans for that trace are sent to Honeycomb regardless of your throughput goals. This means:

• Late spans don't count toward your GoalThroughputPerSec calculations

• A kept trace with 50 spans at decision time + 50 late spans will send 100 spans but only count 50 toward your goal

• Very large traces (100k+ spans) can cause massive quota overruns when most of their spans arrive after the decision

Identifying Late Span Issues

Refinery marks late spans with meta.refinery.send_reason = trace_send_late_span, but only when AddRuleReasonToTrace is set to true in your Refinery configuration. Without that setting, late spans carry no identifying field and this query will return nothing even if late spans are present.

With AddRuleReasonToTrace enabled:

1. Query your usage data filtering by meta.refinery.send_reason = trace_send_late_span

2. Look for spikes in span volume that correlate with late span increases

3. Check if your sampling rules show higher throughput than configured goals

Mitigation Strategies

Adjust Refinery Configuration

You can modify these settings to reduce late spans, but be aware of memory usage implications:

• Increase TraceTimeout: Allows more time for spans to arrive before making sampling decisions on traces without a root span

• Increase SendDelay: Waits longer after the root span arrives before sending the trace, giving more spans time to arrive

• Review SpanLimit: If set, traces exceeding it are decided immediately, and everything after becomes a late span

Note: Higher values increase Refinery's memory usage and may push Refinery into stress relief mode if set too high.

Application-Level Solutions

• Add drop rules for problematic services or trace patterns that generate excessive late spans (late spans of dropped traces are dropped too)

• Filter upstream in your OpenTelemetry collector to drop noisy spans before they reach Refinery

• Improve telemetry design by reducing span cardinality and focusing on meaningful observability data

Infrastructure Improvements

• Ensure stable network connectivity between services and Refinery

• Monitor Refinery pod restarts and stability

• Consider using batching in your telemetry pipeline to reduce network overhead

Current Limitations

Currently, Refinery does not provide a configuration option to drop late-arriving spans for a kept trace. Once a trace is kept, all of its spans will be forwarded to Honeycomb regardless of arrival time. The most effective approach is often to identify and filter out problematic trace patterns at the source or in your collector configuration.