The sky is a terrible network.
A stratospheric balloon telescope produces multi-gigabyte science images. The data comes down through a NASA TDRS relay satellite on a high-gain link that is narrow and lossy, and the uplink budget goes to commands and telemetry, so the ground can't acknowledge every packet the way TCP needs.
The top data rate on the high-gain link.
Half the top rate.
One tenth of the top rate.
Break the link. Watch it recover.
A faithful model of the real daemons: the sender's queue, a lossy link, the ground's bitmap, and the cleaner asking for exactly what's missing. Change the loss, switch the TDRSS High Gain rate, corrupt a packet, or type ctl commands in the terminal. Timings are compressed so a pass takes seconds, not minutes.
Sender · flight auto
Receiver · ground cleaner in 2.5s
32 bytes that answer three questions.
This is a real parachuter packet, encoded byte for byte with a real CRC-32. Hover a byte to learn what it's for. Click one, then flip its bits, and watch the receiver's checks catch the damage. Below, every field is broken out: what it is, how it's set, and how it can differ.
What the receiver checks
One bit per chunk: the ground's to-do list.
For every file in flight the ground keeps 117.incoming (the bytes, each chunk written at its offset) and 117.parmanifest (the bookkeeping). Click chunks to "receive" them and watch the real manifest bytes change.
m31_0142.fits.bz2 · 20 chunks
$ xxd 117.parmanifest
Chunk n lives in bitmap byte n ÷ 8, bit n mod 8, lowest bit first. Each update is written to a temp file and renamed over the old one, so a crash leaves the old or the new manifest, never half of one.
When an outage swallows a whole file.
The cleaner repairs files the ground knows about. But if every packet of a file is lost, the ground never learns it existed, while the payload's ledger says it was sent. Reconciliation closes that gap with no acknowledgements from the ground.
The cleaner repairs it.
At least one packet landed, so the ground has a manifest and knows the file's size. The outage is just one long gap in the bitmap, requested in a single range once the link is back.
Nobody notices. Until now.
No packet landed, so there's no manifest and nothing for the cleaner to chase. The sender already marked the file as sent and never picks it up again. The same happens to files still queued when the sender restarts.
The payload downlinks its ledger
Every 6 h the sender writes files_sent_list_<time>.csv into its first priority folder, so it goes down like any other file: every path, size and queued_at.
Files land at their payload path
Each file lands at final_dir + its full path, and only once it is complete. The folder is the ground's record; no extra log.
The cleaner compares
Every 6 h it reads the newest snapshot and checks each file: landed with the right size, still in flight, still queued, or queued under 3 h ago.
Lost files go again
Anything else was lost whole: it is requested in full, at the back of the queue, at most once per cycle, until it lands.
| file_id | file_name | size | queued_at | verdict |
|---|---|---|---|---|
| 117 | /data/qsc/science/m31_001.fits.bz2 | 5,242,880 | 09:12 | complete |
| 118 | /data/qsc/science/m31_002.fits.bz2 | 5,242,880 | 09:40 | lost whole → resend |
| 119 | /data/qsc/dark/dark_0007.fits.bz2 | 2,097,152 | 10:05 | in flight |
| 120 | /data/qsc/flat/flat_0012.fits.bz2 | 2,097,152 | 13:58 | still queued |
| 121 | /data/qsc/science/m31_003.fits.bz2 | 5,242,880 | 14:51 | too recent (< 3 h) |
Every verdict
holding/; the cleaner is chasing its gaps. Flagged if stuck longer than the grace period.
still queuedWaiting in the sender's live queue.
too recentQueued less than 3 h ago; give it time.
gone on payloadNever arrived and deleted on the payload: reported, can't be resent.
size mismatchLanded, but not at the ledger's size: reported, not resent.
lost wholeRequested again in full, at the back of the queue.
Settings
status_dump_period_secs21600 · ledger snapshot (payload)
reconcile_period_secs21600 · how often to compare (ground)
reconcile_grace_secs10800 · wait before calling a file lost
ctl reconcile-runcompare right now
Worst case with the defaults, a lost file is requested again about 12 h after it was lost: up to 6 h until the next ledger snapshot, then up to 6 h until the next pass. Shorten both periods to find out sooner.
Each pass shrinks the gap by the loss rate.
Extra airtime ≈ p ÷ (1 − p): every chunk costs about one send, plus re-sends only for what was lost. The live demo measured +25.4% at 20% loss.
One binary. Five modes. No restarts.
Checked twice
CRC-32 on every packet and every manifest header. A damaged packet is treated exactly like a lost one.
Power-cut proof
Chunks are synced to disk before their bit is set, and manifests are replaced atomically. Restart and it resumes mid-file.
Retune anything, live
Switch links, change the speed cap or chunk size, pause, or jump the queue over a Unix socket. The TOML config hot-reloads in about a second.
Science first
Ordered priority folders decide what goes next, with a separate debug queue so testing never drains live data.
Multicast-ready
The receiver joins multicast groups by itself, so switching from a unicast to a relay link needs no ground changes.
Every file tracked
A SQLite ledger records every file seen and sent, with CSV snapshots for the ground.
parachuter senderFlight side. Watches folders, chunks, paces, sends.
parachuter receiverGround. Validates, writes, finalises files.
parachuter cleanerGround. Finds gaps, asks for retransmits.
parachuter ctlOperator. Pokes any daemon over its socket.
parachuter monitorLive table of every file in flight.