Plural Telemetry
Logs

Installation

Run a Logs database on Kubernetes: grant it a bucket, declare it as a custom resource, and give each tenant namespace credentials.

This guide assumes the operator is already installed; see Installation. Examples use the telemetry Kubernetes namespace and an S3 bucket named acme-telemetry.

Grant S3 access

Logs authenticates to S3 with EKS Pod Identity. The operator creates a ServiceAccount named after the database, logs, so associate an IAM role with it:

bash
aws eks create-pod-identity-association \
  --cluster-name prod \
  --namespace telemetry \
  --service-account logs \
  --role-arn arn:aws:iam::123456789012:role/plural-telemetry-logs
plural-telemetry-logs policy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::acme-telemetry"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::acme-telemetry/logs/*"
    }
  ]
}

The role's trust policy must allow pods.eks.amazonaws.com to call sts:AssumeRole and sts:TagSession. No keys go in the spec.

Quick start

A Standalone database is a single pod that ingests and queries, which is enough for development and small clusters. Every field not shown takes its default:

logs.yaml
1apiVersion: telemetry.plural.sh/v1alpha12kind: Logs3metadata:41  name: logs52  namespace: telemetry6spec:73  config:8    storage:94      path: logs10      objectStore:115        type: Aws12        aws:136          region: us-east-1147          bucket: acme-telemetry
  1. 1name
    Also names the ServiceAccount the IAM role binds to, and the logs-writer and logs-reader Services (just logs in Standalone mode).
  2. 2namespace
    The Kubernetes namespace the pods run in. Tenant namespaces for your data are separate and come from NamespaceAuthentication resources.
  3. 3config
    Rendered into the server's config file. The operator rolls the pods when it changes.
  4. 4path
    Object-key prefix inside the bucket; shard suffixes are appended. Must match the prefix in the IAM policy. Default logs.
  5. 5type
    Aws, Gcp, Azure, Local or InMemory. With Aws and no key references, the pod uses its Pod Identity credentials.
  6. 6region
    Bucket region. Add endpoint (and allowHTTP if needed) for S3-compatible stores such as MinIO.
  7. 7bucket
    Can be shared by all three databases, as long as their path prefixes differ.

Production

A Sharded database runs writers and readers as separate StatefulSets. Hover a field, or its note, to see what it does:

logs.yaml
1apiVersion: telemetry.plural.sh/v1alpha12kind: Logs3metadata:41  name: logs52  namespace: telemetry6spec:73  mode: Sharded84  config:95    retention: 14d10    storage:116      path: logs12      objectStore:137        type: Aws14        aws:158          region: us-east-1169          bucket: acme-telemetry17    write:1810      durability: applied1911      flushIntervalSeconds: 1020    sharding:2112      leaseDurationSeconds: 152213    segmentDurationSeconds: 360023    request:2414      maxQueryEntries: 50002515      queryConcurrency: 1626    elasticsearch:2716      streamFields: [kubernetes.namespace_name]28  writer:2917    replicas: 33018    resources:31      requests: { cpu: "1", memory: 2Gi }32      limits: { memory: 4Gi }3319    cacheVolume:34      persistentVolumeClaim:35        accessModes: [ReadWriteOnce]36        resources: { requests: { storage: 20Gi } }37  reader:3820    replicas: 23921    cacheVolume:40      emptyDir: { sizeLimit: 20Gi }4122  ingress:42    enabled: true43    hostname: logs.acme.internal44    ingressClass: nginx
  1. 1name
    Also names the ServiceAccount the IAM role binds to, and the logs-writer and logs-reader Services (just logs in Standalone mode).
  2. 2namespace
    The Kubernetes namespace the pods run in. Tenant namespaces for your data are separate and come from NamespaceAuthentication resources.
  3. 3mode
    Standalone (the default) runs one pod that reads and writes. Sharded runs writer and reader StatefulSets that scale independently.
  4. 4config
    Rendered into the server's config file. The operator rolls the pods when it changes.
  5. 5retention
    How long data is kept, counted from ingestion, as 14d, 2w or 36h. Default 14d.
  6. 6path
    Object-key prefix inside the bucket; shard suffixes are appended. Must match the prefix in the IAM policy. Default logs.
  7. 7type
    Aws, Gcp, Azure, Local or InMemory. With Aws and no key references, the pod uses its Pod Identity credentials.
  8. 8region
    Bucket region. Add endpoint (and allowHTTP if needed) for S3-compatible stores such as MinIO.
  9. 9bucket
    Can be shared by all three databases, as long as their path prefixes differ.
  10. 10durability
    When a write is acknowledged: applied (default) once in memory, written once in SlateDB's mutable state, durable once uploaded to object storage.
  11. 11flushIntervalSeconds
    How often writers flush to object storage, and so how far readers can lag behind. Default 10.
  12. 12leaseDurationSeconds
    How long a shard Lease survives without renewal before another writer may take the shard over. Default 15; renewIntervalSeconds (default 5) must be lower.
  13. 13segmentDurationSeconds
    Width of a time partition. Queries prune whole segments by time. Keep it stable for a dataset. Default 3600.
  14. 14maxQueryEntries
    Most log lines one query returns. Default 5000.
  15. 15queryConcurrency
    Concurrent query work per pod. Default 16. maxInFlightQueryBytes (128 MiB) bounds memory.
  16. 16streamFields
    For _bulk ingest: document fields promoted to stream labels. Keep them low-cardinality; other fields become structured metadata.
  17. 17replicas
    Writer count, which is also the storage shard count. Raise it at any time; new shards take writes from the next aligned hour. It cannot be lowered.
  18. 18resources
    Defaults to 250m CPU and 512Mi memory requests with a 2Gi limit. Budget for the write buffer: 64 MiB per shard, plus up to two frozen buffers being flushed.
  19. 19cacheVolume
    Mounted at /var/cache/logs as the disk tier of the block cache (512 MiB RAM and 10 GiB disk by default). A PVC keeps it warm across restarts. Set exactly one of emptyDir or persistentVolumeClaim.
  20. 20replicas
    Readers are stateless; each opens every shard read-only and shares one cache across them. Default 2 in Sharded mode. Scale freely.
  21. 21cacheVolume
    An emptyDir is enough: a restarted reader refills its cache from the bucket.
  22. 22ingress
    Optional. One hostname that routes /write to the writers and /read to the readers. Add tls and pathPrefix as needed.

Apply it with kubectl apply -f logs.yaml. The operator renders the server config, then creates the ServiceAccount, StatefulSets, Services and Ingress. It rolls the pods whenever the spec changes.

Access

Logs serves a tenant namespace only once it has credentials. Each NamespaceAuthentication creates its namespace if needed and grants one username read or write. Create the password Secrets first:

bash
kubectl -n telemetry create secret generic logs-payments-writer \
  --from-literal=password="$(openssl rand -hex 24)"
kubectl -n telemetry create secret generic logs-payments-reader \
  --from-literal=password="$(openssl rand -hex 24)"

Then grant a writer for the collector and a reader for Grafana:

namespace-auth.yaml
1apiVersion: telemetry.plural.sh/v1alpha12kind: NamespaceAuthentication3metadata:4  name: logs-payments-writer5  namespace: telemetry6spec:71  dataStoreRef: { kind: Logs, name: logs }82  namespace: payments93  username: otel-collector104  permission: write115  secretKeyRef: { name: logs-payments-writer, key: password }12---13apiVersion: telemetry.plural.sh/v1alpha114kind: NamespaceAuthentication15metadata:16  name: logs-payments-reader17  namespace: telemetry18spec:19  dataStoreRef: { kind: Logs, name: logs }20  namespace: payments21  username: grafana22  permission: read23  secretKeyRef: { name: logs-payments-reader, key: password }
  1. 1dataStoreRef
    The database this credential is for. One NamespaceAuthentication grants access to exactly one database.
  2. 2namespace
    Tenant namespace. It is created on first use; every route is prefixed with /ns/{namespace}.
  3. 3username
    HTTP basic-auth username. Unique per namespace.
  4. 4permission
    write for ingest routes, read for queries.
  5. 5secretKeyRef
    Secret holding the password, in the same Kubernetes namespace. Rotating it rolls the database's config.

Clients send these as HTTP basic auth.

Connect your tools

Writers serve ingest routes and readers serve queries. Inside the cluster:

ToolURL
Grafana Loki data sourcehttp://logs-reader.telemetry:3100/read/ns/payments
Alloy / Promtail loki.writehttp://logs-writer.telemetry:3100/write/ns/payments/loki/api/v1/push
OTel Collector otlphttp
The exporter appends /v1/logs.
http://logs-writer.telemetry:3100/write/ns/payments/otlp
Fluent Bit / Vector elasticsearch
Speaks the _bulk protocol.
http://logs-writer.telemetry:3100/write/ns/payments/elasticsearch

Any writer accepts any entry and forwards it to the shard's owner, so the plain logs-writer ClusterIP Service is all a client needs. For an OTel Collector and Grafana:

otel-collector.yaml
extensions:
  basicauth/plural:
    client_auth:
      username: otel-collector
      password: ${env:PLURAL_LOGS_PASSWORD}

exporters:
  otlphttp/plural-logs:
    endpoint: http://logs-writer.telemetry:3100/write/ns/payments/otlp
    auth:
      authenticator: basicauth/plural

service:
  extensions: [basicauth/plural]
  pipelines:
    logs:
      exporters: [otlphttp/plural-logs]
grafana-datasource.yaml
apiVersion: 1
datasources:
  - name: Plural Logs
    type: loki
    url: http://logs-reader.telemetry:3100/read/ns/payments
    basicAuth: true
    basicAuthUser: grafana
    secureJsonData:
      basicAuthPassword: $PLURAL_LOGS_READ_PASSWORD
Elasticsearch shippers

Fluent Bit, Vector, Fluentd and Filebeat can use /write/ns/payments/elasticsearch as their Elasticsearch host. Promote low-cardinality fields to stream labels with spec.config.elasticsearch.streamFields, or per request with _stream_fields. See the API reference.

Verify

bash
kubectl -n telemetry get logs
# NAME   MODE      READY   AGE
# logs   Sharded   True    2m

kubectl -n telemetry port-forward svc/logs-reader 3100 &
curl -u grafana:"$PASSWORD" http://localhost:3100/read/ns/payments/loki/api/v1/labels

An empty data array is expected until the collector has sent something.

Scaling

  • Writers: raise spec.writer.replicas. New pods start first, and the new shards take writes from the next aligned hour. Existing data stays where it is, so scale-down is blocked until it expires. See epoch sharding.
  • Readers: change spec.reader.replicas at any time. They hold no state beyond their cache.
  • Heavy queries: raise request.queryConcurrency and request.maxInFlightQueryBytes together with reader memory.
ArchitectureHow Logs compares with Loki.API referenceEvery LogQL and ingest route, with examples.BenchmarksLatency against Loki, run by run.