Streaming to analytics systems

Stream your ledger data in real-time to analytics systems like ClickHouse, Elasticsearch, or custom HTTP endpoints. This lets you build dashboards, enable full-text search, or trigger external workflows—without polling the API.

Architecture#

The export system has two components: exporters define where data goes (driver + connection config), and pipelines connect a ledger to an exporter. Multiple ledgers can share the same exporter.

What gets streamed#

Every change to your ledger is recorded as a log entry. Pipelines stream these logs to your chosen destination:

Quick start#

Here's the complete flow to start streaming data:

1. Create an exporter (defines where data goes)

curl -X POST $FORMANCE_API_URL/api/ledger/v2/_/exporters \
  -H "Content-Type: application/json" \
  -d '{
    "driver": "clickhouse",
    "config": {
      "dsn": "clickhouse://localhost:9000"
    }
  }'

Response:

{
  "data": {
    "id": "exp-abc123",
    "driver": "clickhouse",
    "config": { "dsn": "clickhouse://localhost:9000" },
    "createdAt": "2025-01-15T10:00:00Z"
  }
}

2. Create a pipeline (connects a ledger to the exporter)

curl -X POST $FORMANCE_API_URL/api/ledger/v2/my-ledger/pipelines \
  -H "Content-Type: application/json" \
  -d '{
    "exporterID": "exp-abc123"
  }'

Pipelines start automatically. Logs begin streaming immediately.

Exporter drivers#

Your endpoint receives a JSON array of log entries:

[
  {
    "ledger": "my-ledger",
    "id": 42,
    "type": "NEW_TRANSACTION",
    "date": "2025-01-15T10:30:00Z",
    "data": {
      "transaction": {
        "id": 1,
        "postings": [...],
        "metadata": {}
      }
    }
  }
]

Return a 2xx status code to acknowledge receipt.

Managing exporters#

List all exporters:

curl -X GET $FORMANCE_API_URL/api/ledger/v2/_/exporters

Get a specific exporter:

curl -X GET $FORMANCE_API_URL/api/ledger/v2/_/exporters/<EXPORTER_ID>

Delete an exporter:

curl -X DELETE $FORMANCE_API_URL/api/ledger/v2/_/exporters/<EXPORTER_ID>

Managing pipelines#

Check pipeline status#

curl -X GET $FORMANCE_API_URL/api/ledger/v2/my-ledger/pipelines/<PIPELINE_ID>

The lastLogID shows how far the pipeline has progressed:

{
  "data": {
    "id": "pipe-xyz789",
    "ledger": "my-ledger",
    "exporterID": "exp-abc123",
    "lastLogID": 1042,
    "enabled": true,
    "createdAt": "2025-01-15T10:00:00Z"
  }
}

Stop and restart#

Pipelines remember their position. Stop and start without losing progress:

Stop streaming:

curl -X POST $FORMANCE_API_URL/api/ledger/v2/my-ledger/pipelines/<PIPELINE_ID>/stop

Resume streaming:

curl -X POST $FORMANCE_API_URL/api/ledger/v2/my-ledger/pipelines/<PIPELINE_ID>/start

Replay from the beginning#

Reset the pipeline to re-stream all logs:

curl -X POST $FORMANCE_API_URL/api/ledger/v2/my-ledger/pipelines/<PIPELINE_ID>/reset

Delete a pipeline#

curl -X DELETE $FORMANCE_API_URL/api/ledger/v2/my-ledger/pipelines/<PIPELINE_ID>

Scaling with multiple pipelines#

You can attach multiple ledgers to the same exporter. This helps avoid rate limits on external systems, spread load, and maintain independent data flows per ledger.

curl -X POST $FORMANCE_API_URL/api/ledger/v2/ledger-001/pipelines \
  -H "Content-Type: application/json" \
  -d '{
    "exporterID": "<EXPORTER_ID>"
  }'

Repeat for each ledger — all pipelines share the same exporter.