Ledger v2.3 — Real-Time Logs Exporter & Pipelines

Overview

Starting with Ledger 2.3, you can now export transaction logs in real time to external systems using the new Logs Exporter & Pipelines feature.
This allows you to stream ledger logs to external sinks like ClickHouse, Elasticsearch, or HTTP endpoints for analytics, observability, or integration purposes.

Supported exporters include:

• clickhouse

• elasticsearch

• http

• stdout

How It Works

The export system is composed of two main components:

1. Exporters – define where and how to send data.
   (e.g., connection info, driver type)

2. Pipelines – define which ledger(s) send data to which exporter(s).
   (each pipeline connects one ledger to one exporter)

Step 1: Create an Exporter

An exporter defines the destination and configuration for exported data.

Example – Create a ClickHouse exporter:

curl -X POST /api/ledger/v2/_/exporters -d '{
  "driver": "clickhouse",
  "config": { 
    "dsn": "clickhouse://default:default@clickhouse:9000/ledger" 
  }
}'

Response:

{ 
  "data": {
    "id": "3fec8cd5-0bee-4cc5-8578-a71d97e39b58",
    "createdAt": "2025-10-17T09:47:59.077604Z",
    "driver": "clickhouse",
    "config": { 
        "dsn": "clickhouse://default:default@clickhouse:9000/ledger" 
    } 
  } 
}

This creates an exporter that writes logs to a ClickHouse instance.

Step 2: Create a Pipeline for a Ledger

A pipeline connects a ledger to an exporter. Once created, all new transaction logs from that ledger will be exported through the specified exporter.

Example – Create a pipeline for ledger-002:

curl -X POST /api/ledger/v2/ledger-002/pipelines -d '{ 
  "exporterID": "3fec8cd5-0bee-4cc5-8578-a71d97e39b58"
}'

Response:

{
  "data": {
    "ledger": "ledger-002",
    "exporterID": "3fec8cd5-0bee-4cc5-8578-a71d97e39b58",
    "createdAt": "2025-10-17T09:52:25.648245Z",
    "id": "6ddb1f75-ec41-469a-a8d2-82b775e3512c",
    "enabled": true 
  } 
}

Now, all transaction events from ledger-002 will be exported in real time to ClickHouse.

Managing Throughput and Rate Limiting

You can attach multiple ledgers (pipelines) to the same exporter.
This design allows you to control throughput and parallelism while avoiding rate limits on your external systems.

Why this matters:

• Some external sinks (e.g., Elasticsearch, HTTP APIs) may rate-limit incoming writes.

• Instead of one pipeline handling a massive volume of transactions, you can spread load across multiple pipelines—each corresponding to a ledger.

• This allows for better concurrency, load balancing, and isolation between ledgers.

Example Scenario:

Suppose you operate multiple ledgers:

• ledger-001

• ledger-002

• ledger-003

You can connect each to the same ClickHouse exporter:

curl -X POST /api/ledger/v2/ledger-001/pipelines -d '{"exporterID": "<exporter-id>"}'
curl -X POST /api/ledger/v2/ledger-002/pipelines -d '{"exporterID": "<exporter-id>"}'
curl -X POST /api/ledger/v2/ledger-003/pipelines -d '{"exporterID": "<exporter-id>"}'

Each ledger will maintain its own export pipeline, ensuring data flows independently while sharing the same destination configuration.

This helps:

• Prevent overload of a single export stream.

• Simplify scaling strategies.

• Centralize monitoring of export targets.

Summary