# Kubernetes

> Install the Fixter collector Helm chart to ship pod logs, Kubernetes events, and cluster and host metrics, then point your workloads at it for their own traces and logs.

On Kubernetes, the shortest route into Fixter is the Fixter collector: a standard
OpenTelemetry Collector distribution, packaged as a Helm chart. Installing it gets your
pod logs, Kubernetes events, and cluster and host metrics into Fixter with no application
changes at all. Your own services' traces and logs are added on top, by pointing them at
the collector instead of at Fixter directly.

This page covers the case where you want Fixter to run a collector for you. If you
already run an OpenTelemetry Collector, keep it and add a Fixter exporter instead: see
[Send data with OpenTelemetry](/setup/opentelemetry/).

## Before you start

You need an API key. Get one from
[Settings > API keys](https://app.fixter.dev/settings/api-keys) in the app, or ask an
agent connected over [MCP](/setup/mcp/) for `get_ingestion_credentials`, which returns a
one-time link you open in a browser to reveal the key.

Kubernetes 1.23 or later, and `kubectl` and `helm` pointed at the cluster you want to
monitor.

Install into a namespace that is not under a `restricted` Pod Security Admission policy.
The log-collecting agent mounts host paths and runs as root, both of which a restricted
namespace rejects. A plain namespace created as below is fine.

## Install

Create the namespace and put the key in a Secret first, so it never reaches your shell
history or the pod's process list:

```bash
kubectl create namespace fixter
kubectl create secret generic fixter-credentials \
  --namespace fixter \
  --from-literal=api-key=<your-key>
```

Then install the chart pointing at that Secret:

```bash
helm install fixter-collector \
  oci://ghcr.io/fixter-dev/charts/fixter-collector \
  --namespace fixter \
  --set fixter.existingSecret=fixter-credentials \
  --set fixter.existingSecretKey=api-key \
  --set fixter.clusterName=<your-cluster>
```

The endpoint already defaults to `https://ingest.fixter.dev`, so there is nothing else
you have to set. Pin a version with `--version <x.y.z>` for reproducible deploys; without
it you get the latest published chart.

The chart also accepts `--set fixter.apiKey=<your-key>` directly, which is mutually
exclusive with `existingSecret`. It is the faster way to try this on a scratch cluster
and the wrong way to run it anywhere real.

If a GitOps or Terraform system already manages your Helm releases, the
[Claude Code plugin](/setup/claude-code-plugin/) handles that case rather than you
running the command by hand.

## Check the key immediately

**A wrong API key fails silently.** The collector starts, reports healthy, and ships
nothing. Run the bundled test straight after installing:

```bash
helm test fixter-collector -n fixter
```

It posts an empty metrics payload and checks the response, which proves the key is valid
and Fixter is reachable. It does not prove every export succeeds, so also confirm data is
arriving on the [Logs page](https://app.fixter.dev/logs) within a few minutes.

## What gets installed

Two workloads:

| Component | Kind | Collects |
| --- | --- | --- |
| `agent` | DaemonSet, one per node | kubelet and host metrics, pod logs, and an OTLP relay on ports 4317 and 4318 |
| `cluster` | Deployment, one replica | cluster-state metrics, Kubernetes events, Prometheus targets |

The `cluster` deployment runs exactly one replica and that is not configurable. Its
receivers each produce a full copy of their output, so a second replica would double
count everything.

**Kubernetes events arrive as logs**, one per event as it happens. They carry the reason
a pod changed state, `OOMKilled`, `CrashLoopBackOff`, `FailedScheduling`, `Unhealthy`,
which cluster metrics cannot express on their own. Volume is low on a healthy cluster and
rises during an incident, which is when you want them. Turn them off with
`--set cluster.events=false`.

**Name every cluster.** `fixter.clusterName` is not detected automatically, and an unset
name means the cluster is simply not identified in your telemetry. That is survivable
with one cluster and ambiguous the moment you have two, which is why it is in the install
command above.

## Send your applications' telemetry through it

The agent exposes an OTLP endpoint on every node. Point your workloads at its Service,
named `<release>-fixter-collector-agent`, which for the install above is
`fixter-collector-agent`. Set these on your application deployments:

```
OTEL_EXPORTER_OTLP_ENDPOINT=http://fixter-collector-agent.fixter.svc.cluster.local:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
```

Traffic stays on the node it started from. Your applications still need OpenTelemetry
instrumentation to produce traces and logs in the first place: see
[Send data with OpenTelemetry](/setup/opentelemetry/) for that half.

**Service names are filled in, never overwritten.** Pod logs and kubelet metrics carry no
service identity of their own, so the agent supplies one from Kubernetes metadata: the
deployment name (falling back to the container name), the namespace, and the pod name.
Telemetry your applications send keeps whatever `service.name` they set. Node and host
metrics stay serviceless, which is correct for infrastructure.

## Log volume

Pod logs are collected by default. `kube-system` and the collector's own pods are already
excluded. Before you leave this running, do the arithmetic: roughly 216-byte JSON lines
at 10,000 logs per second is about 2.2 MB/s, or **186 GB a day**.

Trim by namespace or pod. Put this in a `values.yaml` next to wherever you keep your Helm
configuration:

```yaml
# values.yaml
agent:
  logs:
    excludeNamespaces: [kube-system, istio-system]
    excludePods: [chatty-app]
```

Apply it, keeping the settings from your original install:

```bash
helm upgrade fixter-collector \
  oci://ghcr.io/fixter-dev/charts/fixter-collector \
  --namespace fixter --reuse-values -f values.yaml
```

Or turn pod logs off entirely and rely on your applications exporting their own:

```bash
helm upgrade fixter-collector \
  oci://ghcr.io/fixter-dev/charts/fixter-collector \
  --namespace fixter --reuse-values \
  --set agent.logs.enabled=false
```

## How log severity is worked out

A pod log line carries no severity of its own, and a stack trace arrives as one line per
frame. What the collector can do about that depends on whether it recognises the format.

Out of the box it reads only what a line's own shape reveals, and does not guess:

| Format | Severity | Record boundaries |
| --- | --- | --- |
| JSON with a string level (zap, winston, logback) | yes, from `level` | one per line |
| JSON with a numeric level (pino, bunyan) | yes, `10` to `60` mapped | one per line |
| glog and klog, used by Kubernetes components | yes, from the `I`/`W`/`E`/`F` prefix | one per line |
| any other text format | none | one per line |

That last row is deliberate. Guessing a level from an unknown format goes wrong in ways
that are worse than having no level: a service named `trace-service` read as TRACE, a
successful nginx line containing `/error/` read as ERROR. Joining continuation lines is
riskier still, because a wrong join merges unrelated events into one record and buries an
ERROR inside an INFO. Nothing is ever dropped for being unparseable; an unrecognised line
still reaches Fixter, just without a severity.

**This is the practical argument for logging JSON from your own services.** Do that and
severity, stack traces, and trace correlation all work with no configuration.

Common datastores are handled automatically, because they log under predictable container
and pod names: ClickHouse, Doris, PostgreSQL, MySQL, Kafka, and the AWS VPC CNI. Turn one
off with `agent.logs.builtinFormats.mysql.enabled: false` in your values file.

For a text format the collector does not know, point a preset at the pods that emit it,
in the same `values.yaml` and applied the same way:

```yaml
# values.yaml
agent:
  logs:
    formats:
      - preset: spring
        include: ["/var/log/pods/prod_*/*/*.log"]
      - preset: python
        include: ["/var/log/pods/*_worker-*/*/*.log"]
```

Presets available: `spring`, `clickhouse`, `postgres`, `mysql`, `kafka`, `doris-fe`,
`logfmt`, `python`, `dotnet`, `go-stdlib`, and `zap-console`.

Two things to know before writing one:

- **Formats are selected by log file path**, in the shape
  `/var/log/pods/<namespace>_<pod>_<uid>/<container>/0.log`, so you target a namespace, a
  pod name prefix, or a container. Pod labels and annotations are not available at the
  point the line is parsed.
- **First match wins**, and each format automatically excludes the paths of every format
  above it. Put narrow patterns before broad ones, and do not add those exclusions by
  hand: two receivers reading one file duplicates every record.

## Scraping Prometheus endpoints

If services already expose Prometheus metrics, the cluster collector can scrape them.
There are two ways to say so in your values file, and you can use both at once.

**A list of targets**, when you know the addresses:

```yaml
# values.yaml
integrations:
  prometheus:
    targets:
      - job: my-service
        endpoints: ["my-service.default.svc.cluster.local:9090"]
        interval: 30s
```

**A raw `scrape_configs` block**, when you need service discovery or relabeling. The
collector ships the full upstream Prometheus receiver, and what you put here is passed
to it verbatim, so any config from your existing Prometheus setup works unchanged:

```yaml
# values.yaml
integrations:
  prometheus:
    scrapeConfigs:
      - job_name: kubernetes-pods
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: "true"
```

Either key enables the receiver. Apply it the same way as any other values change:

```bash
helm upgrade fixter-collector \
  oci://ghcr.io/fixter-dev/charts/fixter-collector \
  --namespace fixter --reuse-values -f values.yaml
```

The cluster collector restarts with the new scrape config, and the scraped series show
up on the [Metrics page](https://app.fixter.dev/metrics) under the service the targets
belong to.

## What this does not collect

Worth knowing before you conclude something is broken:

- **Control plane logs on managed Kubernetes.** On EKS, apiserver, etcd, scheduler and
  controller logs go to CloudWatch rather than to pods, and kubelet and containerd log to
  journald on the node. The collector reads neither.
- **`kube-system` pods** are excluded by default for volume. CoreDNS, Karpenter,
  external-dns and the CSI and load balancer controllers do write pod logs, so if you
  want those, set `agent.logs.excludeNamespaces: []` and accept the extra volume.
- **Datastore metrics.** Datastore logs are parsed, as above, but there are no metrics
  receivers for ClickHouse, Doris, PostgreSQL, MySQL or Redis yet.
- **Managed cloud services** whose telemetry lives in CloudWatch rather than in your
  cluster, such as RDS. For AWS logs and metrics that reach a Firehose stream, see
  [AWS Firehose](/setup/aws-firehose/).

## If nothing shows up

- **The pods are Ready but no data arrives.** Almost always the API key. Run
  `helm test fixter-collector -n fixter`; a failure there is a bad or wrong key.
- **The agent pod will not schedule.** The namespace is likely under a `restricted` Pod
  Security Admission policy. The agent needs read-only host path mounts for
  `/var/log/pods` and host metrics, so install it into a namespace that permits them.
- **Metrics arrive but no pod logs.** Check `agent.logs.enabled` and your namespace and
  pod exclusions. Note that `kube-system` is excluded by default.
- **Logs arrive with no severity, or stack traces split across records.** The format is
  not one the collector recognises. Log JSON, or point a preset at those pods, as above.
- **You cannot tell two clusters apart.** `fixter.clusterName` is not detected
  automatically; set it on each install.

Still stuck? Email [info@fixter.dev](mailto:info@fixter.dev) and one of our founders will help you fix your problem asap.

## Related

- [Send data with OpenTelemetry](/setup/opentelemetry/): instrumenting your applications,
  and the exporter settings if you already run your own collector
- [Telemetry](/using-fixter/telemetry/): what each signal answers and why you want all three
- [Data model](/reference/data-model/): the shape of what arrives