Installation

This guide assumes the operator is already installed; see Installation. Examples use the telemetry Kubernetes namespace and an S3 bucket named acme-telemetry.

Grant S3 access

Metrics authenticates to S3 with EKS Pod Identity. The operator creates a ServiceAccount named after the database, metrics, so associate an IAM role with it:

aws eks create-pod-identity-association \
  --cluster-name prod \
  --namespace telemetry \
  --service-account metrics \
  --role-arn arn:aws:iam::123456789012:role/plural-telemetry-metrics

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::acme-telemetry"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::acme-telemetry/metrics/*"
    }
  ]
}

The role's trust policy must allow pods.eks.amazonaws.com to call sts:AssumeRole and sts:TagSession. No keys go in the spec.

Quick start

A Standalone database is a single pod that ingests and queries, which is enough for development and small clusters. Every field not shown takes its default, including 60 days of retention:

1apiVersion: telemetry.plural.sh/v1alpha12kind: Metrics3metadata:41  name: metrics52  namespace: telemetry6spec:73  config:8    storage:94      path: metrics10      objectStore:115        type: Aws12        aws:136          region: us-east-1147          bucket: acme-telemetry

Production

A Sharded database runs writers and readers as separate StatefulSets. Hover a field, or its note, to see what it does:

1apiVersion: telemetry.plural.sh/v1alpha12kind: Metrics3metadata:41  name: metrics52  namespace: telemetry6spec:73  mode: Sharded84  config:95    retention: 60d10    storage:116      path: metrics12      objectStore:137        type: Aws14        aws:158          region: us-east-1169          bucket: acme-telemetry17    write:1810      durability: applied1911      flushIntervalSeconds: 1020    sharding:2112      leaseDurationSeconds: 1522    request:2313      maxRequestBytes: 3355443224  writer:2514    replicas: 22615    resources:27      requests: { cpu: "1", memory: 2Gi }28      limits: { memory: 4Gi }2916    cacheVolume:30      persistentVolumeClaim:31        accessModes: [ReadWriteOnce]32        resources: { requests: { storage: 20Gi } }33  reader:3417    replicas: 23518    cacheVolume:36      emptyDir: { sizeLimit: 20Gi }3719  ingress:38    enabled: true39    hostname: metrics.acme.internal40    ingressClass: nginx

Apply it with kubectl apply -f metrics.yaml. The operator renders the server config, then creates the ServiceAccount, StatefulSets, Services and Ingress. It rolls the pods whenever the spec changes.

Access

Metrics serves a tenant namespace only once it has credentials. Each NamespaceAuthentication creates its namespace if needed and grants one username read or write. Create the password Secrets first:

kubectl -n telemetry create secret generic metrics-payments-writer \
  --from-literal=password="$(openssl rand -hex 24)"
kubectl -n telemetry create secret generic metrics-payments-reader \
  --from-literal=password="$(openssl rand -hex 24)"

Then grant a writer for the collector and a reader for Grafana:

1apiVersion: telemetry.plural.sh/v1alpha12kind: NamespaceAuthentication3metadata:4  name: metrics-payments-writer5  namespace: telemetry6spec:71  dataStoreRef: { kind: Metrics, name: metrics }82  namespace: payments93  username: otel-collector104  permission: write115  secretKeyRef: { name: metrics-payments-writer, key: password }12---13apiVersion: telemetry.plural.sh/v1alpha114kind: NamespaceAuthentication15metadata:16  name: metrics-payments-reader17  namespace: telemetry18spec:19  dataStoreRef: { kind: Metrics, name: metrics }20  namespace: payments21  username: grafana22  permission: read23  secretKeyRef: { name: metrics-payments-reader, key: password }

Clients send these as HTTP basic auth.

Connect your tools

Writers serve ingest routes and readers serve queries. Inside the cluster:

Any writer accepts any sample and forwards it to the shard's owner, so the plain metrics-writer ClusterIP Service is all a client needs. For an OTel Collector and Grafana:

extensions:
  basicauth/plural:
    client_auth:
      username: otel-collector
      password: ${env:PLURAL_METRICS_PASSWORD}

exporters:
  otlphttp/plural-metrics:
    endpoint: http://metrics-writer.telemetry:8080/write/ns/payments
    auth:
      authenticator: basicauth/plural

service:
  extensions: [basicauth/plural]
  pipelines:
    metrics:
      exporters: [otlphttp/plural-metrics]

apiVersion: 1
datasources:
  - name: Plural Metrics
    type: prometheus
    url: http://metrics-reader.telemetry:8080/read/ns/payments
    basicAuth: true
    basicAuthUser: grafana
    secureJsonData:
      basicAuthPassword: $PLURAL_METRICS_READ_PASSWORD

For Prometheus or Alloy, add a remote_write entry with the writer URL above and the same basic_auth credentials.

Verify

kubectl -n telemetry get metrics
# NAME   MODE      READY   AGE
# metrics Sharded   True    2m

kubectl -n telemetry port-forward svc/metrics-reader 8080 &
curl -u grafana:"$PASSWORD" http://localhost:8080/read/ns/payments/api/v1/labels

Scaling

• Writers: raise spec.writer.replicas. New pods start first, and the new shards take samples from the next aligned hour. Series written across the cutover appear in both shards and are joined at query time. Scale-down is blocked. See epoch sharding.
• Readers: change spec.reader.replicas at any time. They hold no state beyond their cache.