When using Honeycomb Refinery to manage trace sampling, you may wonder why a particular trace is being included. This typically comes down to how your Refinery rules are configured and how they are evaluated against incoming traces.

You notice that a trace is being processed according to a specific rule in your Refinery configuration, but it's not immediately clear why that rule was matched. Understanding the rule evaluation process and how to debug it is essential for effective sampling and troubleshooting.

First, ensure you have AddRuleReasonToTrace configured to true in your config:

RefineryTelemetry:
  AddRuleReasonToTrace: true

https://github.com/honeycombio/refinery/blob/main/config_complete.yaml#L242

Once this is enabled, the meta.refinery.reason field will be available in Honeycomb queries and will show the rule that the span matched. Once you know the specific rule, it’s easier to confirm against the rules file why it matched.

Things to check in your rules file:

• Rules are evaluated top-down. They should be ordered most > least specific. If a trace matches a rule, it will not be evaluated against any further down the list

• Rules can be scoped to trace or span

  • Rules scoped to trace, the conditions only need to be met across all spans in a trace

  • Rules scoped to span, the conditions must match on a single span or span event!

• Refinery makes decisions for the entire trace, if a rule is matched on a span or trace (depending on the scope) then all spans in that trace make the same decision

• Rules are scoped to environments, default evaluates for all environments

• ALL conditions need to be met in order for a trace to match the rule

• Field names and values are case sensitive

• Check the data type e.g if you matching on http.response_code => 500, this field should be a number. If it's stored in Honeycomb as a string, it may not match as you expect

Related links:

Rule-based sampler Docs

Still need help?

Contact our Support team at https://support.honeycomb.io/