Forward logs to Loki, OpenSearch or Splunk

The monitoring bundle can run Grafana Alloy next to its Grafana. Alloy reads the logs of the containers on the Docker host, labels each line with the container it came from, and sends it to your log platform. That's the place for security monitoring and retention: the bundle's dashboards show runs, not container logs.

Platform	`LOG_FORWARD_SINK`	How Alloy sends
Grafana Loki, Grafana Cloud Logs	`loki`	Loki push API
OpenSearch	`opensearch`	OpenTelemetry (OTLP over gRPC) to OpenSearch Data Prepper or Amazon OpenSearch Ingestion
Splunk	`splunk`	HTTP Event Collector (HEC)

Alloy has no direct OpenSearch writer, so OpenSearch logs go through Data Prepper, the ingestion service OpenSearch provides. If you don't run one, set up an otel_logs_source pipeline first (see below).

Before you start

The monitoring bundle, 2026.12 or later, on each Docker host whose containers you want forwarded. Hop and Putki containers on another host need Alloy on that host as well.
The address of your platform, and a login for it: a Loki user and token, a Data Prepper user, or a Splunk HEC token.
Outbound access from the host to that address.

Set it up

Add the settings for your platform to .env, next to docker-compose-monitoring.yml.

Loki:

LOG_FORWARD_SINK=loki
LOG_FORWARD_URL=https://loki.example.com/loki/api/v1/push
LOG_FORWARD_USER=123456             # leave empty if Loki needs no login
LOG_FORWARD_PASSWORD=glc_...
LOG_FORWARD_TENANT=                 # X-Scope-OrgID, for a multi-tenant Loki

OpenSearch (through Data Prepper):

LOG_FORWARD_SINK=opensearch
LOG_FORWARD_URL=data-prepper.example.com:21892
LOG_FORWARD_USER=putki              # Data Prepper's http_basic user, if it has one
LOG_FORWARD_PASSWORD=...

Splunk:

LOG_FORWARD_SINK=splunk
LOG_FORWARD_URL=https://splunk.example.com:8088/services/collector
LOG_FORWARD_PASSWORD=<HEC token>
LOG_FORWARD_INDEX=putki             # leave empty for the token's default index

Name this host, so you can tell hosts apart in the platform, and limit what is sent if the host runs other containers:

LOG_FORWARD_HOST=hop-prod-01
LOG_FORWARD_CONTAINERS=putki.*|hop.*   # regular expression on the container name; default all

Start the overlay with the bundle:
```
docker compose -f docker-compose-monitoring.yml -f docker-compose-monitoring.logs.yml up -d
```
On a host without the dashboards, the overlay runs on its own: docker compose -f docker-compose-monitoring.logs.yml up -d.

Check that it works

docker compose -f docker-compose-monitoring.yml -f docker-compose-monitoring.logs.yml logs alloy shows Alloy's log. Lines with level=error name the component that failed, for example loki.write.siem with the HTTP status the platform answered.

Then search for the lines in your platform:

Platform	Search
Loki	`{job="putki-log-forwarding", host="hop-prod-01"}`
OpenSearch	Documents where `log.attributes.job` is `putki-log-forwarding` (the field names depend on your Data Prepper pipeline)
Splunk	`sourcetype="putki:container" job="putki-log-forwarding"`

What arrives

Each log line becomes one entry, with these labels (Loki), attributes (OpenSearch) or indexed fields (Splunk):

Label	Value
`container`	Container name, for example `putki-web`
`compose_project`, `compose_service`	Docker Compose project and service, when the container has them
`image`	The container's image
`host`	`LOG_FORWARD_HOST`, or Alloy's container id when it is not set
`job`	`putki-log-forwarding`

In Splunk, the event's own host is unknown and its source is the token's default: the version of Alloy in the bundle cannot set them from labels. Search on the indexed fields instead, for example sourcetype="putki:container" container="putki-web".

The line itself is sent as Docker recorded it. Hop writes one line per log message; a Java stack trace arrives as several lines.

Things to know

The first start sends what the containers already logged. Alloy then remembers how far it read each container, in the alloy_data volume, and resumes there after a restart. A platform may reject the oldest lines of that first batch as too old; Loki does by default. Alloy logs that and carries on.
Short scheduled jobs. Scheduled Hop jobs run in containers that are removed when they end. Alloy looks for new containers every 5 seconds, so a job that is over sooner may not be forwarded. Its run is still in the execution log.
Access to Docker. Alloy reads logs through the Docker socket. Mounting it read-only does not limit what the Docker API allows, so treat the Alloy container as having control of the host, as with any collector that reads the socket.
While the platform is unreachable, Alloy retries and keeps the lines in memory; a restart in that time loses them.
Splunk needs one of Alloy's community components, its HEC exporter. The overlay starts Alloy with --feature.community-components.enabled for it.
TLS with a private CA: mount the CA file into the alloy service and add a tls block with ca_file to the sink's endpoint in its .alloy file (Alloy reference).

OpenSearch: a Data Prepper pipeline

A minimal Data Prepper pipeline that takes what Alloy sends and writes it to an index. Adjust the host, credentials and index to your cluster:

putki-logs:
  source:
    otel_logs_source:
      port: 21892
      ssl: true
      sslKeyCertChainFile: /usr/share/data-prepper/certs/server.crt
      sslKeyFile: /usr/share/data-prepper/certs/server.key
      authentication:
        http_basic:
          username: putki
          password: <password>
  sink:
    - opensearch:
        hosts: ["https://opensearch.example.com:9200"]
        username: <opensearch user>
        password: <password>
        index: putki-logs-%{yyyy.MM.dd}

Alloy always connects with TLS, because gRPC does not send a password over a plain connection. For a test pipeline with ssl: false, edit alloy/opensearch.alloy: delete the auth line and set insecure = true in its tls block.

Amazon OpenSearch Ingestion signs requests with AWS SigV4 instead of a password: in alloy/opensearch.alloy, replace otelcol.auth.basic with otelcol.auth.sigv4 and point LOG_FORWARD_URL at the pipeline's endpoint.

Send somewhere else

The configuration is two files in the bundle's alloy/ folder: docker.alloy chooses the containers and labels, and one file per platform sends them on. To send elsewhere, write your own platform file defining loki.process "sink", put it in alloy/ and set LOG_FORWARD_SINK to its name. Alloy can also write to syslog, Amazon S3 and any OTLP endpoint; see the Alloy components reference.

Settings

See the reference for every LOG_FORWARD_* setting.

Before you start​

Set it up​

Check that it works​

What arrives​

Things to know​

OpenSearch: a Data Prepper pipeline​

Send somewhere else​

Settings​