Plural Telemetry
Traces

Installation

Run a Traces database on Kubernetes: grant it a bucket, declare it as a custom resource, and give each tenant namespace credentials.

This guide assumes the operator is already installed; see Installation. Examples use the telemetry Kubernetes namespace and an S3 bucket named acme-telemetry.

Grant S3 access

Traces authenticates to S3 with EKS Pod Identity. The operator creates a ServiceAccount named after the database, traces, so associate an IAM role with it:

bash
aws eks create-pod-identity-association \
  --cluster-name prod \
  --namespace telemetry \
  --service-account traces \
  --role-arn arn:aws:iam::123456789012:role/plural-telemetry-traces
plural-telemetry-traces policy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::acme-telemetry"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::acme-telemetry/traces/*"
    }
  ]
}

The role's trust policy must allow pods.eks.amazonaws.com to call sts:AssumeRole and sts:TagSession. No keys go in the spec.

Quick start

A Standalone database is a single pod that ingests and queries, which is enough for development and small clusters. Every field not shown takes its default:

traces.yaml
1apiVersion: telemetry.plural.sh/v1alpha12kind: Traces3metadata:41  name: traces52  namespace: telemetry6spec:73  config:8    storage:94      path: traces10      objectStore:115        type: Aws12        aws:136          region: us-east-1147          bucket: acme-telemetry
  1. 1name
    Also names the ServiceAccount the IAM role binds to, and the traces-writer and traces-reader Services (just traces in Standalone mode).
  2. 2namespace
    The Kubernetes namespace the pods run in. Tenant namespaces for your data are separate and come from NamespaceAuthentication resources.
  3. 3config
    Rendered into the server's config file. The operator rolls the pods when it changes.
  4. 4path
    Object-key prefix inside the bucket; shard suffixes are appended. Must match the prefix in the IAM policy. Default traces.
  5. 5type
    Aws, Gcp, Azure, Local or InMemory. With Aws and no key references, the pod uses its Pod Identity credentials.
  6. 6region
    Bucket region. Add endpoint (and allowHTTP if needed) for S3-compatible stores such as MinIO.
  7. 7bucket
    Can be shared by all three databases, as long as their path prefixes differ.

Production

A Sharded database runs writers and readers as separate StatefulSets. Hover a field, or its note, to see what it does:

traces.yaml
1apiVersion: telemetry.plural.sh/v1alpha12kind: Traces3metadata:41  name: traces52  namespace: telemetry6spec:73  mode: Sharded84  config:95    retention: 14d10    storage:116      path: traces12      objectStore:137        type: Aws14        aws:158          region: us-east-1169          bucket: acme-telemetry17    write:1810      durability: applied1911      flushIntervalSeconds: 1020    sharding:2112      leaseDurationSeconds: 152213    segmentDurationSeconds: 360023    request:2414      maxCandidates: 100002515      maxSpansPerTrace: 10000026  writer:2716    replicas: 32817    resources:29      requests: { cpu: "1", memory: 2Gi }30      limits: { memory: 4Gi }3118    cacheVolume:32      persistentVolumeClaim:33        accessModes: [ReadWriteOnce]34        resources: { requests: { storage: 20Gi } }35  reader:3619    replicas: 23720    cacheVolume:38      emptyDir: { sizeLimit: 20Gi }3921  ingress:40    enabled: true41    hostname: traces.acme.internal42    ingressClass: nginx
  1. 1name
    Also names the ServiceAccount the IAM role binds to, and the traces-writer and traces-reader Services (just traces in Standalone mode).
  2. 2namespace
    The Kubernetes namespace the pods run in. Tenant namespaces for your data are separate and come from NamespaceAuthentication resources.
  3. 3mode
    Standalone (the default) runs one pod that reads and writes. Sharded runs writer and reader StatefulSets that scale independently.
  4. 4config
    Rendered into the server's config file. The operator rolls the pods when it changes.
  5. 5retention
    How long data is kept, counted from ingestion, as 14d, 2w or 36h. Default 14d.
  6. 6path
    Object-key prefix inside the bucket; shard suffixes are appended. Must match the prefix in the IAM policy. Default traces.
  7. 7type
    Aws, Gcp, Azure, Local or InMemory. With Aws and no key references, the pod uses its Pod Identity credentials.
  8. 8region
    Bucket region. Add endpoint (and allowHTTP if needed) for S3-compatible stores such as MinIO.
  9. 9bucket
    Can be shared by all three databases, as long as their path prefixes differ.
  10. 10durability
    When a write is acknowledged: applied (default) once in memory, written once in SlateDB's mutable state, durable once uploaded to object storage.
  11. 11flushIntervalSeconds
    How often writers flush to object storage, and so how far readers can lag behind. Default 10.
  12. 12leaseDurationSeconds
    How long a shard Lease survives without renewal before another writer may take the shard over. Default 15; renewIntervalSeconds (default 5) must be lower.
  13. 13segmentDurationSeconds
    Width of a time partition. Searches prune whole segments by time. Default 3600.
  14. 14maxCandidates
    Most traces one TraceQL search will decode and evaluate. Default 10000.
  15. 15maxSpansPerTrace
    Safety limit while assembling a trace by ID. Default 100000.
  16. 16replicas
    Writer count, which is also the storage shard count. Raise it at any time; new shards take writes from the next aligned hour. It cannot be lowered.
  17. 17resources
    Defaults to 250m CPU and 512Mi memory requests with a 2Gi limit. Budget for the write buffer: 64 MiB per shard, plus up to two frozen buffers being flushed.
  18. 18cacheVolume
    Mounted at /var/cache/traces as the disk tier of the block cache (512 MiB RAM and 10 GiB disk by default). A PVC keeps it warm across restarts. Set exactly one of emptyDir or persistentVolumeClaim.
  19. 19replicas
    Readers are stateless; each opens every shard read-only and shares one cache across them. Default 2 in Sharded mode. Scale freely.
  20. 20cacheVolume
    An emptyDir is enough: a restarted reader refills its cache from the bucket.
  21. 21ingress
    Optional. One hostname that routes /write to the writers and /read to the readers. Add tls and pathPrefix as needed.

Apply it with kubectl apply -f traces.yaml. The operator renders the server config, then creates the ServiceAccount, StatefulSets, Services and Ingress. It rolls the pods whenever the spec changes.

Access

Traces serves a tenant namespace only once it has credentials. Each NamespaceAuthentication creates its namespace if needed and grants one username read or write. Create the password Secrets first:

bash
kubectl -n telemetry create secret generic traces-payments-writer \
  --from-literal=password="$(openssl rand -hex 24)"
kubectl -n telemetry create secret generic traces-payments-reader \
  --from-literal=password="$(openssl rand -hex 24)"

Then grant a writer for the collector and a reader for Grafana:

namespace-auth.yaml
1apiVersion: telemetry.plural.sh/v1alpha12kind: NamespaceAuthentication3metadata:4  name: traces-payments-writer5  namespace: telemetry6spec:71  dataStoreRef: { kind: Traces, name: traces }82  namespace: payments93  username: otel-collector104  permission: write115  secretKeyRef: { name: traces-payments-writer, key: password }12---13apiVersion: telemetry.plural.sh/v1alpha114kind: NamespaceAuthentication15metadata:16  name: traces-payments-reader17  namespace: telemetry18spec:19  dataStoreRef: { kind: Traces, name: traces }20  namespace: payments21  username: grafana22  permission: read23  secretKeyRef: { name: traces-payments-reader, key: password }
  1. 1dataStoreRef
    The database this credential is for. One NamespaceAuthentication grants access to exactly one database.
  2. 2namespace
    Tenant namespace. It is created on first use; every route is prefixed with /ns/{namespace}.
  3. 3username
    HTTP basic-auth username. Unique per namespace.
  4. 4permission
    write for ingest routes, read for queries.
  5. 5secretKeyRef
    Secret holding the password, in the same Kubernetes namespace. Rotating it rolls the database's config.

Clients send these as HTTP basic auth.

Connect your tools

Writers serve ingest routes and readers serve queries. Inside the cluster:

ToolURL
Grafana Tempo data sourcehttp://traces-reader.telemetry:3200/read/ns/payments
OTel Collector otlphttp
The exporter appends /v1/traces.
http://traces-writer.telemetry:3200/write/ns/payments
Zipkin v2 reportershttp://traces-writer.telemetry:3200/write/ns/payments/api/v2/spans

Any writer accepts any span and forwards each trace to its shard's owner, so the plain traces-writer ClusterIP Service is all a client needs. For an OTel Collector and Grafana:

otel-collector.yaml
extensions:
  basicauth/plural:
    client_auth:
      username: otel-collector
      password: ${env:PLURAL_TRACES_PASSWORD}

exporters:
  otlphttp/plural-traces:
    endpoint: http://traces-writer.telemetry:3200/write/ns/payments
    auth:
      authenticator: basicauth/plural

service:
  extensions: [basicauth/plural]
  pipelines:
    traces:
      exporters: [otlphttp/plural-traces]
grafana-datasource.yaml
apiVersion: 1
datasources:
  - name: Plural Traces
    type: tempo
    url: http://traces-reader.telemetry:3200/read/ns/payments
    basicAuth: true
    basicAuthUser: grafana
    secureJsonData:
      basicAuthPassword: $PLURAL_TRACES_READ_PASSWORD
OTLP/gRPC and Jaeger

Writer pods also listen for OTLP/gRPC on 4317 and Jaeger gRPC on 14250, but the operator's Service exposes only HTTP and internal gRPC. Expose those ports with your own Service, and pass the tenant namespace in x-scope-orgid metadata alongside basic auth.

Verify

bash
kubectl -n telemetry get traces
# NAME   MODE      READY   AGE
# traces Sharded   True    2m

kubectl -n telemetry port-forward svc/traces-reader 3200 &
curl -u grafana:"$PASSWORD" http://localhost:3200/read/ns/payments/api/v2/search/tags

Scaling

  • Writers: raise spec.writer.replicas. New pods start first, and the new shards take traces from the next aligned hour, by each trace's earliest span. Scale-down is blocked. See epoch sharding.
  • Readers: change spec.reader.replicas at any time. They hold no state beyond their cache.
  • Large traces: raise request.maxSpansPerTrace and reader memory together.
ArchitectureHow Traces compares with Tempo.API referenceEvery TraceQL and ingest route, with examples.BenchmarksLatency against Tempo, run by run.