#How it works

A short tour of the pipeline. The full engineering detail lives in ARCHITECTURE.md in the repository.

#The pipeline

Source file
  -> decode + resize + BGR conversion        (Python/OpenCV or Rust/FFmpeg)
  -> ASCII map / adaptive encode / raw / DCT  (server)
  -> session WebSocket packets
  -> ordered JavaScript reconstruction        (browser)
  -> Canvas text, or BGR-to-RGBA ImageData

Source audio -> /audio FFmpeg process -> audio element -> playback clock

Three separate costs sit in that chain: decoding the source, encoding into ASCILINE's representation, and reconstructing it in the browser. They are measured and tuned separately; speeding up one does not speed up the others.

#Components

Part Role
Server (stream_server.py) FastAPI endpoints, source queue configuration, WebSocket sessions, pacing
Engines Python by default; an optional Rust engine for decode, mapping and encoding
Browser client Receives packets, reconstructs frames in order, draws to Canvas
SDK (src/) AsciiPlayer, LiveSession and the <ascf-player> element
Compiler Turns a video into a static .ascf file

#The adaptive codec

For ASCII color modes the server can pick the smallest of four lossless encodings for each frame and tag it with one byte. The tag byte also has a fifth value, DCT, shown detached at the bottom because it is not one of the per-frame candidates:

TagEncodingBest for
0RAWFrames that don't compress
1ZLIBGeneral motion
2DELTAOnly the cells that changed
3RLE_FULLLarge flat-color regions
4DCTLossy pixel compression, a separate codec for pixel mode

Clients opt in with /ws?codec=adaptive. Without it they get the original protocol, byte for byte, so older clients keep working. A keyframe is forced periodically so late joiners can resync.

Tag 4, DCT, is the lossy codec for pixel mode. Live streams use it with --pixel-codec dct (the client also asks for pixel_codec=dct-v1); static files get it from compiler.py --profile. The next section covers it.

#Pixel mode and the DCT codec

Raw pixel mode sends three bytes per cell. At 450×253 and 60 FPS that is roughly 20 MB/s, which is why it needs help.

The DCT profile converts to Y/Cb/Cr, predicts each frame from the previous reconstruction, transforms and quantizes the residual, and compresses the result. It is lossy (even at quality 100) and applies only to pixel mode with file playback. The server and browser keep matching predictor state per connection, and reset it on seek.

#The optional Rust engine

Python is the default and needs no build. An optional Rust engine handles decoding, mapping and DCT encoding natively and is what reaches 60 FPS at larger pixel grids. It must be built locally; it is Windows-tested, and Linux/macOS native builds are not yet validated. The Docker image runs the Python server only.

--engine rust fails loudly if the module is missing. --engine auto falls back to Python and logs why.

#Handshake and sync

The client sends an INIT message with FPS, mode and grid size; the server replies with the stream. For synchronized playback the server sends four preroll frames and waits for the client's playback-ready, anchoring to the audio clock. A seek sends a request ID, the server repositions its decoder, and a SEEKED marker separates the old timeline from the new one so stale frames never reach the screen.

#Static .ascf files

The same representation can be written to a file: a small binary header, then length-prefixed packets. Playback needs no ASCILINE backend. See Static player.

#Where to look for more