LogWisp

Pipeline-based log transport: tail, filter, format, fan out

In development Pre-1.0. On this site it streams the ViF multiplayer server log.

GoFreeBSDLinuxDockerObservabilityNetworkingTCPHTTPSSETLSmTLSArgon2idSCRAMTOMLJSONPipelinesHot ReloadWeb UI
See it liveGitHub

LogWisp started as a way to stop SSH-ing into servers to tail and grep log files (the story). It is now one static binary, lw, that runs anything from a shell filter to a multi-node aggregation topology.

Pipelines
Each pipeline is independent: sources feed a flow that rate-limits, filters and formats, and the flow fans out to any number of sinks.
Never blocks
A full queue drops and counts its own entries rather than stalling the stages around it, so a slow reader costs only itself.
Chaining
Links carry the structured entry, not formatted text, so a relay can filter and reformat as if the entries were local, and every entry keeps the node it came from.
Security
TLS 1.2/1.3 everywhere, mutual TLS with identity-based authorization, and Argon2id-SCRAM passwords. lw tls makes the certificates, or a listener makes its own at startup; unknown configuration keys are refused.
Access
Every listener can refuse addresses by allow and deny lists before the TLS handshake, cap each client's connections (and over HTTP its request rate), and read the client's address from a load balancer's PROXY header.
Viewer
A browser at an HTTP sink gets a live page that follows the stream, with a search and a level filter; a sink that keeps a backlog shows the latest entries first. Behind a site's TLS proxy it signs in with SCRAM.
Reload
Edit the TOML and the service rebuilds itself. A candidate that fails validation leaves the running pipelines untouched.
Filter
With no file and no options, lw is a Unix filter: stdin to stdout, line for line.
How a pipeline worksSee it live

Every plugin LogWisp ships, in one pipeline. A configuration picks any number of sources and sinks; the flow stages are each optional and run in this order.

Sources

  • file Tails every matching file in a directory, through rotation
  • console Reads stdin; the source when none is given
  • tcp_chainlistens Entries from another node over one persistent stream
  • http_chainlistens Entries from another node in batches over POST
  • random Generates random logs records for testing

Flow

  1. rate_limit Token bucket, and a cap on entry size
  2. filters Include and exclude RE2 patterns, in order
  3. format raw, txt or json, with a sanitizer policy
  4. heartbeat A keep-alive entry on a timer

Sinks

  • console stdout or stderr, levels in colour
  • file Rotating files with size caps and retention
  • httplistens Server-Sent Events and status; a browser viewer with search
  • tcplistens Formatted lines broadcast to every client
  • tcp_chaindials To the next node; holds entries across reconnects
  • http_chaindials To the next node in batches, with retry
listens accepts connections, dials makes them; every listener takes address rules and per-client limits. A chain sink on one node feeds a chain source on the next.
Entries
A source turns each line into a structured entry: time, level, message, optional JSON fields, and the node it came from. The formatter renders it for the sinks, which also receive the entry itself.
Drops
Every queue is bounded. A full one drops the entry for that stage, sink or client alone and counts it. The console sink waits instead, so a filter loses no line to a slow reader; the TCP chain sink holds a line across reconnects, so an outage downstream surfaces as counted drops upstream, not silent loss.
Topologies
An edge node tails local files and dials an aggregator; the aggregator listens for its edges and serves one stream, or writes one archive. Presets build both: lw --preset edge and lw --preset aggregator.

The ViF multiplayer lobby streams every game server’s log through one LogWisp pipeline. Open the Server log panel there and host a game to watch it fill.

From

ViF game serversEach writes JSON lines to a log volume on its node

Sources

  • file Tails the volume's *.jsonl files, each line as written

Flow

  1. rate_limit 400 entries a second, bursts of 800
  2. filters Excludes TRACE records and admission chatter
  3. format raw: the JSON line, byte for byte

Sinks

  • httplistens Server-Sent Events on loopback

To

The lobby pageRelayed by the game allocator, served by nginx at /vif/api/logs
Open the live log

Runtime

  • Platforms: Linux and FreeBSD, natively or in a container
  • Toolchain: Go 1.27, no CGO, one static binary
  • Dependencies: own Go modules for configuration, logging, TOML, SCRAM, colour and terminal output
  • Configuration: a TOML file, or the same keys as flags and environment
  • Presets: pipe, tail, serve, edge and aggregator
  • License: BSD-3-Clause

Build

Clone
git clone https://github.com/lixenwraith/logwisp --depth 1
Build
cd logwisp && make build
Run
./bin/lw --preset tail,path=/var/log/app
Deploy
make image builds a scratch image that runs read-only with no capabilities. Natively, deploy/lw-deploy.sh sets up an edge, an aggregator or a standalone node as a systemd or rc.d service.

Quick start

# a filter: errors and warnings only
tail -F app.log | ./bin/lw --filter include,patterns=ERROR,patterns=WARN

# stdin as a live SSE stream, with a browser viewer at http://127.0.0.1:8080/
journalctl -f | ./bin/lw --sink http,host=127.0.0.1,port=8080

# a directory's logs, served to one subnet
./bin/lw --preset serve,path=/var/log/app,listen=0.0.0.0:8080,allow=192.0.2.0/24

# any command line as a configuration file
./bin/lw --preset serve,path=/var/log/app --dump > logwisp.toml