AWS Firehose

Some of your telemetry never passes through code you control. RDS, load balancers, Lambda and the rest of AWS write to CloudWatch, where no OpenTelemetry SDK can reach them. Fixter accepts an Amazon Data Firehose HTTP delivery stream, so that data can land in the same place as everything else.

This is for AWS-managed sources only. Your own services should send OpenTelemetry directly, which gives you traces as well as logs and metrics.

What you can send

Source	Fixter endpoint	Arrives as
CloudWatch metric streams	https://ingest.fixter.dev/aws/firehose/metrics	metrics
CloudWatch Logs, via a subscription filter	https://ingest.fixter.dev/aws/firehose/logs	logs

Both are Firehose HTTP endpoint destinations. Point a Firehose stream at the URL and set its access key to your Fixter API key.

Before you start

You need a Fixter API key, from Settings > API keys. Firehose sends it as the access key on every request, so it is the whole authentication story: no IAM role on the Fixter side and no other header.

Set up with one click

A CloudFormation stack sends the log groups and metrics of one AWS account and region to Fixter. Your API key is the only thing you have to fill in.

1. Sign in to AWS and switch to the account and region whose telemetry you want in Fixter.

2. Open the stack. You can read the template first.

   Launch stack

3. Paste your Fixter API key, tick the box acknowledging that CloudFormation creates IAM resources, and choose Create stack. It takes a few minutes. Leave AWS’s Permissions - optional section empty: CloudFormation then acts with your own permissions.

4. Confirm. Log lines appear on the Logs page and metrics on the Metrics page within a few minutes.

Logs and metrics each have a dropdown, and both cover what you create later.

Logs start on AWS service logs only (recommended): the log groups under /aws/, such as RDS, without Lambda and Container Insights. Your own apps’ log groups, like the /ecs/... groups of ECS services, are left out, because an app that also sends its logs to Fixter with OpenTelemetry would otherwise arrive twice. The stack filters inside your AWS account, so those lines never leave it. Choose All log groups when your apps do not send their own logs to Fixter and CloudWatch is the only place they go. Log groups to leave out, under Advanced, takes full log group names, comma separated, for anything else you do not want sent.

Metrics start on All except EC2, EBS and Usage (recommended): those three namespaces are often the largest share of an account’s metrics, and a Fixter collector on your hosts or cluster already reports the same machines.

Choosing All namespaces can be a lot. It is billable volume on both sides: AWS charges for each metric update the stream sends, and Fixter for what arrives. In one of our own accounts, all namespaces came to about twelve times the volume of AWS/RDS alone, and the default to about seven times.

To pick individual namespaces, create the stack with the default, then narrow it in AWS, which has a tick list of the namespaces your account really has. Open CloudWatch > Metrics > Streams, select the stream the stack created, choose Edit and select the namespaces to include. If you update the stack later, check the stream’s namespaces again.

A stack covers one region, so launch it again in each region you use. To stop sending, delete the stack.

Unless logs are set to None, the stack fails if the account already has an account-level subscription filter policy in that region, because AWS allows one. In that case, or when you want only some log groups or metric namespaces, do the setup by hand as described in the next two sections.

Send CloudWatch metrics

1. Create a Firehose stream in the Amazon Data Firehose console, source Direct PUT, destination HTTP Endpoint.

2. Configure the endpoint. Set the URL to https://ingest.fixter.dev/aws/firehose/metrics and paste your Fixter API key into the access key field. Leave content encoding on GZIP; Fixter decompresses either way.

3. Create a CloudWatch metric stream (CloudWatch > Metrics > Streams), choose the Firehose stream from step 1, select OpenTelemetry 1.0 as the output format, and pick the namespaces you want. Start narrow, for example AWS/RDS and AWS/ApplicationELB; every namespace you add is billable volume on both sides.

4. Confirm. Within a few minutes the metrics appear on the Metrics page, and list_metrics over MCP returns them.

Send CloudWatch Logs

Logs need three things in AWS: a Firehose stream that delivers to Fixter, an IAM role that lets CloudWatch Logs write into that stream, and a subscription filter on each log group. CloudWatch has no “log stream” wizard like the one for metric streams, so the Firehose stream is created in the Firehose console, not in CloudWatch. It is a separate stream from the metrics one, because the URL differs.

1. Create the Firehose stream. Open the Amazon Data Firehose console and choose Create Firehose stream.

   • Source: Direct PUT
   • Destination: HTTP Endpoint
   • HTTP endpoint URL: https://ingest.fixter.dev/aws/firehose/logs
   • Authentication: Use access key, and paste your Fixter API key
   • Content encoding: GZIP
   • Backup settings: pick an S3 bucket. AWS requires one for records it cannot deliver.

   If the console offers Decompress source records from Amazon CloudWatch Logs, leave it off. Fixter expects the records the way CloudWatch Logs writes them, compressed, and skips anything else.

2. Create an IAM role for CloudWatch Logs. In IAM, create a role with this trust policy:

   {
     "Version": "2012-10-17",
     "Statement": [{
       "Effect": "Allow",
       "Principal": { "Service": "logs.amazonaws.com" },
       "Action": "sts:AssumeRole"
     }]
   }

   and this permission policy. Both can be pasted as they are:

   {
     "Version": "2012-10-17",
     "Statement": [{
       "Effect": "Allow",
       "Action": ["firehose:PutRecord", "firehose:PutRecordBatch"],
       "Resource": "arn:aws:firehose:*:*:deliverystream/*"
     }]
   }

   This lets CloudWatch Logs write to any Firehose stream in the account. To limit it to one stream, replace the Resource value with that stream’s ARN, shown on the stream’s page in the Firehose console.

3. Subscribe a log group. In CloudWatch > Log groups, open the log group, go to the Subscription filters tab and choose Create > Create Amazon Data Firehose subscription filter. Select the stream from step 1 and the role from step 2, leave the filter pattern empty to send every line, name the filter and start streaming. Repeat this step for each log group you want in Fixter; they can all share the one stream and role.

4. Confirm. Within a few minutes the lines appear on the Logs page.

How the logs are labelled

Service is the last path segment of the log group name:

Log group	Service
/aws/lambda/checkout-worker	checkout-worker
/aws/rds/instance/orders-prod/postgresql	orders-prod
/aws/rds/cluster/orders-prod/postgresql	postgresql
/aws/eks/prod/cluster	cluster

RDS instances are the one exception: they use the instance name, which keeps one database’s logs together instead of splitting them by log type. Aurora cluster log groups and everything else follow the last-segment rule, so filter those by log group instead.

Level is read from PostgreSQL-style prefixes (ERROR:, FATAL:, WARNING:, LOG:). Lines without one, including Lambda output, arrive with an empty level.

Each line also carries where it came from, filterable in QuerySQL:

Field	Value
resource.aws.log.group	the full log group name
resource.aws.log.stream	the log stream, also available as source_instance_id
resource.cloud.account.id	the AWS account id
resource.cloud.provider	aws

If nothing shows up

• Firehose reports delivery failures. The access key is wrong. It is your Fixter API key, pasted whole, with no Bearer prefix. Firehose retries into its S3 backup bucket, so check there for what it could not deliver.
• The stream succeeds but nothing is queryable. Fixter skips records it cannot decode rather than rejecting the whole batch, which keeps one bad record from stalling a stream. The usual cause on metrics is the wrong output format: a metric stream must be OpenTelemetry 1.0, not JSON. On logs it is Firehose decompression: turn off Decompress source records from Amazon CloudWatch Logs on the stream.
• The subscription filter cannot be created, or the stream receives nothing. The IAM role is missing or does not allow firehose:PutRecord on that stream.
• Logs arrive under a service name you did not expect. That is the log group’s last path segment, as above. Filter by resource.aws.log.group when the segment is too generic.
• You are looking for control plane or kubelet logs from EKS. Those live in CloudWatch and journald respectively, so a log group subscription is exactly how you reach the CloudWatch half. See Kubernetes for what the collector covers instead.

Still stuck? Email info@fixter.dev and one of our founders will help you fix your problem asap.

Related

• Send data with OpenTelemetry: the main path, for your own services
• Kubernetes: cluster and pod telemetry
• Data model: the shape of what arrives