When dealing with large traces that contain an excessive number of spans, it can be beneficial to sample or drop these traces to prevent performance issues and optimize your observability data. This article explains how to use Honeycomb Refinery to sample traces based on their span count.

Using NUM_DESCENDANTS for early sampling decisions

To make sampling decisions based on the number of spans in a trace, even before the root span is received, you can use the ?.NUM_DESCENDANTS virtual field in your Refinery rules. This approach is particularly useful for long-running traces where the root span may not be available for an extended period.

Here's how to implement this in your Refinery configuration:

  1. Open your Refinery configuration file.

  2. In the rules section, add a condition using the ?.NUM_DESCENDANTS field.

  3. Set a threshold for the maximum number of spans you want to allow.

Example rule:

rules:
  - name: "Drop large traces"
    condition:
      field: "?.NUM_DESCENDANTS"
      operator: ">"
      value: 5000
    samplerType: "static"
    rate: 0

This rule will drop all traces with more than 5,000 spans. Adjust the threshold according to your specific needs.

Alternative: Using AddSpanCountToRoot

For cases where you can wait for the root span, you can use the AddSpanCountToRoot setting in Refinery. This option adds a meta.span_count field to the root span, which you can then use in your sampling rules.

  1. Enable the AddSpanCountToRoot setting in your Refinery configuration.

  2. Create a rule using the meta.span_count field.

For more information on configuring AddSpanCountToRoot, refer to the Honeycomb documentation.

Additional Resources

For the most up-to-date and comprehensive information on Refinery configuration and virtual fields, consider checking both the official Honeycomb documentation and the GitHub repository:

Note that the GitHub repository may sometimes contain more recent information or advanced features that haven't yet been incorporated into the official documentation.