A Rust crate and command-line tool, first flown on the SuperBIT balloon telescope

parachuter

Reliable, rate-limited file downlink over lossy, one-way UDP links. Every byte comes down, and nothing is sent twice unless it was lost.

Watch it work, live ↓
32 Bheader on every packet
1 bitper chunk to track a file
+25%airtime at 20% packet loss
0restarts to retune a link
§ 1The problem

The sky is a terrible network.

A stratospheric balloon telescope produces multi-gigabyte science images. The data comes down through a NASA TDRS relay satellite on a high-gain link that is narrow and lossy, and the uplink budget goes to commands and telemetry, so the ground can't acknowledge every packet the way TCP needs.

1000 kbps
TDRSS High Gain (Fastest)

The top data rate on the high-gain link.

500 kbps
TDRSS High Gain (Fast)

Half the top rate.

100 kbps
TDRSS High Gain (Slow)

One tenth of the top rate.

Fire and forget.The sender streams every chunk once, blind. It can't wait for acknowledgements that may never come.
Track, don't trust.The ground keeps one bit per chunk and knows exactly what is missing, even after a power cut.
Ask for the gaps.A low-rate side channel requests only the missing chunk ranges. Nothing that arrived is ever sent again.
§ 2Live simulator

Break the link. Watch it recover.

A faithful model of the real daemons: the sender's queue, a lossy link, the ground's bitmap, and the cleaner asking for exactly what's missing. Change the loss, switch the TDRSS High Gain rate, corrupt a packet, or type ctl commands in the terminal. Timings are compressed so a pass takes seconds, not minutes.

Sender · flight auto

Queue (front first)
Priority folders (walked top first)
TDRSS High Gain (Slow)100 kbpsloss 10%

Receiver · ground cleaner in 2.5s

Reconcilernext in 18s
waiting for the first ledger snapshot to come down
downloads/
nothing yet
0.0ssim time
0packets sent
0lost on link
0CRC rejects
0retransmits
0recovered whole
0%extra airtime
0files landed
Event lognewest first
    Operator terminaltry: ctl status · help
    parachuter ctl, connected to the simulated sender. Type help for commands.
    $
    § 3Anatomy of a packet

    32 bytes that answer three questions.

    This is a real parachuter packet, encoded byte for byte with a real CRC-32. Hover a byte to learn what it's for. Click one, then flip its bits, and watch the receiver's checks catch the damage. Below, every field is broken out: what it is, how it's set, and how it can differ.

    Identifyis it ours?
    Locatewhere does it go?
    Verifyis it intact?
    Payload · 32 of up to 16,160 bytes shown · bzip2-compressed FITS data

    What the receiver checks

    § 4The manifest

    One bit per chunk: the ground's to-do list.

    For every file in flight the ground keeps 117.incoming (the bytes, each chunk written at its offset) and 117.parmanifest (the bookkeeping). Click chunks to "receive" them and watch the real manifest bytes change.

    m31_0142.fits.bz2 · 20 chunks

    Click a chunk to mark it received or lost.

    $ xxd 117.parmanifest

    Chunk n lives in bitmap byte n ÷ 8, bit n mod 8, lowest bit first. Each update is written to a temp file and renamed over the old one, so a crash leaves the old or the new manifest, never half of one.

    § 5Files lost whole · new in v0.2

    When an outage swallows a whole file.

    The cleaner repairs files the ground knows about. But if every packet of a file is lost, the ground never learns it existed, while the payload's ledger says it was sent. Reconciliation closes that gap with no acknowledgements from the ground.

    Outage cuts through a file

    The cleaner repairs it.

    At least one packet landed, so the ground has a manifest and knows the file's size. The outage is just one long gap in the bitmap, requested in a single range once the link is back.

    Outage swallows every packet

    Nobody notices. Until now.

    No packet landed, so there's no manifest and nothing for the cleaner to chase. The sender already marked the file as sent and never picks it up again. The same happens to files still queued when the sender restarts.

    1

    The payload downlinks its ledger

    Every 6 h the sender writes files_sent_list_<time>.csv into its first priority folder, so it goes down like any other file: every path, size and queued_at.

    2

    Files land at their payload path

    Each file lands at final_dir + its full path, and only once it is complete. The folder is the ground's record; no extra log.

    3

    The cleaner compares

    Every 6 h it reads the newest snapshot and checks each file: landed with the right size, still in flight, still queued, or queued under 3 h ago.

    4

    Lost files go again

    Anything else was lost whole: it is requested in full, at the back of the queue, at most once per cycle, until it lands.

    On the payload/data/qsc/science/m31_002.fits.bz2 On the ground/downloaded_files/data/qsc/science/m31_002.fits.bz2
    downloaded_files/…/status_dumps/files_sent_list_1790376000.csvreconciler verdict →
    file_idfile_namesizequeued_atverdict
    117/data/qsc/science/m31_001.fits.bz25,242,88009:12complete
    118/data/qsc/science/m31_002.fits.bz25,242,88009:40lost whole → resend
    119/data/qsc/dark/dark_0007.fits.bz22,097,15210:05in flight
    120/data/qsc/flat/flat_0012.fits.bz22,097,15213:58still queued
    121/data/qsc/science/m31_003.fits.bz25,242,88014:51too recent (< 3 h)

    Every verdict

    completeAt its mirrored path with the ledger's size. in flightPartial, in holding/; the cleaner is chasing its gaps. Flagged if stuck longer than the grace period. still queuedWaiting in the sender's live queue. too recentQueued less than 3 h ago; give it time. gone on payloadNever arrived and deleted on the payload: reported, can't be resent. size mismatchLanded, but not at the ledger's size: reported, not resent. lost wholeRequested again in full, at the back of the queue.

    Settings

    status_dump_period_secs21600 · ledger snapshot (payload) reconcile_period_secs21600 · how often to compare (ground) reconcile_grace_secs10800 · wait before calling a file lost ctl reconcile-runcompare right now

    Worst case with the defaults, a lost file is requested again about 12 h after it was lost: up to 6 h until the next ledger snapshot, then up to 6 h until the next pass. Shorten both periods to find out sooner.

    § 6Why it converges

    Each pass shrinks the gap by the loss rate.

    missing(k) ≈ N × pk
    5passes to finish
    +25%extra airtime
    155missing after pass 1

    Extra airtime ≈ p ÷ (1 − p): every chunk costs about one send, plus re-sends only for what was lost. The live demo measured +25.4% at 20% loss.

    § 7Built for a flight you can't go and fix

    One binary. Five modes. No restarts.

    Checked twice

    CRC-32 on every packet and every manifest header. A damaged packet is treated exactly like a lost one.

    Power-cut proof

    Chunks are synced to disk before their bit is set, and manifests are replaced atomically. Restart and it resumes mid-file.

    Retune anything, live

    Switch links, change the speed cap or chunk size, pause, or jump the queue over a Unix socket. The TOML config hot-reloads in about a second.

    Science first

    Ordered priority folders decide what goes next, with a separate debug queue so testing never drains live data.

    Multicast-ready

    The receiver joins multicast groups by itself, so switching from a unicast to a relay link needs no ground changes.

    Every file tracked

    A SQLite ledger records every file seen and sent, with CSV snapshots for the ground.

    parachuter sender

    Flight side. Watches folders, chunks, paces, sends.

    parachuter receiver

    Ground. Validates, writes, finalises files.

    parachuter cleaner

    Ground. Finds gaps, asks for retransmits.

    parachuter ctl

    Operator. Pokes any daemon over its socket.

    parachuter monitor

    Live table of every file in flight.

    crates.io ↗ docs.rs ↗