From 13 Mbps to Beating Pion: How We Made webrtc-rs Data Channels Fast (with AI on a Short Leash)
This is a guest post by Stefano Di Martino (StefanoD), who led the data-channel performance work tracked in issue #101 โ eighteen merged PRs across
webrtcandrtcthat took webrtc-rs data-channel throughput from a 44ร deficit against Pion to a decisive lead. This post covers both the optimizations and the AI-assisted workflow that produced them. We're delighted to share it here.
When issue #101 was opened, the numbers were
brutal. The exact same data-channels-flow-control example ran at ~570 Mbps on Pion and
~13 Mbps on webrtc-rs. A 44ร gap on a data channel โ the one part of WebRTC where you'd hope a
systems language would shine.
That gap is gone. On my machine today, webrtc-rs data-channel throughput beats Pion's โ decisively
in multi-connection aggregate (roughly 2โ3ร), and single-connection too once you enable the opt-in
dedicated reactor thread. And for the same bytes transferred, webrtc-rs burns 50โ75% fewer CPU
cycles, instructions, and cache/branch misses than Pion across every case I measured โ that
per-byte efficiency is the most robust part of the result. I'll show the full poop tables below,
including the case where webrtc-rs's plain default still loses to Pion, because that nuance is the
honest version of the story. The maintainer
asked me whether I'd write
up how we got here. This is that write-up.
I want to be honest about one thing up front, because it's the more interesting story: most of
this work was done with an AI coding agent. Not "I asked a chatbot for tips" โ the agent read the
code, formed hypotheses, wrote the patches, and drafted the PRs. What made that produce real,
mergeable, correct optimizations instead of plausible-sounding garbage was the harness I built
around it: a set of hard, external ground-truth checks the AI was not allowed to argue with.
poop, perf, and bpftrace weren't just how I measured โ they were how I kept the AI honest.
Let me tell both stories at once: what we changed, and how we worked.
The Rule That Mattered Most: Measure, Don't Reason
An AI agent is a fluent, confident hypothesis machine. Ask it why some code is slow and it will give you a beautifully-argued answer. It might even be right. The problem is you can't tell the difference between "right" and "confidently wrong" from the prose โ and in performance work, confidently wrong is the default. The received wisdom about what's slow is almost always out of date or was measured on someone else's hardware.
So the first harness was simple: no optimization is real until poop says it is.
poop ("Performance Optimizer Observation Platform") runs two
binaries head-to-head and reports the deltas in wall-time, instructions, cycles, cache-misses,
branch-misses and peak RSS โ with statistical significance markers, so it tells you when a
difference is just noise. Every single change in this project had to survive an A/B poop run
against the exact commit before it. If the agent claimed "this removes an allocation on the hot
path," fine โ prove it: build both, poop them, show me the instruction-count delta and a perf
profile where that allocation symbol actually disappears.
This killed a lot of good-sounding ideas. It also produced the single most useful discovery of the whole effort, which I'll get to.
A concrete example of the discipline: early on, the agent "knew" (from training data) that
sendmmsg/UDP GSO batching was the obvious win for UDP throughput. It sounded right. I made it
measure instead. It instrumented the driver's send path and found that on the 1 KB-message
benchmark, 83% of send bursts were a single packet โ GSO had nothing to batch. We declined
the optimization and wrote down why. (Weeks later, once other changes had started coalescing sends,
GSO did become a win โ but only because we had the measurement to know when the world had changed.
More on that below.)
The Second Rule: The Loopback Lies to You
The very first thing perf told us contradicted the existing diagnosis. There was a handoff
document from earlier macOS investigation describing a 90-second linear throughput ramp and an SCTP
receive-window ceiling. On Linux, none of it reproduced. perf showed the machine was ~90% idle โ
only ~1.6 of 16 cores busy. This wasn't a bandwidth problem. It was a latency problem: the
in-process ping-pong between the application task and the driver task was spending tens of
milliseconds per round trip in the tokio scheduler, not in SCTP.
That distinction โ latency-bound vs CPU-bound vs memory-bound โ turned out to be the thing you have to get right before any number means anything, and it's exactly where a single benchmark will fool you. So the third harness was A/B testing across deliberately different benchmark regimes, each stressing a different resource:
- Single-connection loopback โ latency-bound. Great for finding scheduling and round-trip overhead; useless for measuring per-byte CPU wins (the CPU is mostly idle, so a 12% instruction cut shows up as ~0% throughput).
- N-connection aggregate (10+ pairs) โ CPU-bound once the cores saturate. This is where per-byte efficiency (allocations, copies, crypto, hashing) actually shows up as throughput.
- Flood / bulk-transfer โ send-bound with buffers under pressure. This is where memory (RSS) and send back-pressure behavior show up, and where a slow allocator hurts.
- Large-message vs 1 KB-message โ changes whether syscall batching (GSO) can engage at all.
The number of times a change looked like a win in one regime and a wash (or a regression) in another is the reason this list exists. A 12% instruction reduction that's "sub-noise on throughput" is still worth shipping for an SFU that runs thousands of connections and pays the energy bill โ but you only know that if you measure both the single-connection wall-clock and the CPU-bound aggregate. "Instruction-count deltas overstate wall-clock when the removed work was high-IPC" is now a rule I have tattooed on my brain.
The Third Rule: Never Quote the RFC from Memory
This is the harness I'd recommend to anyone doing protocol work with an AI, and it's the one that caught the most subtle bug.
LLMs will happily "cite" RFC 4960 ยง6.2 or tell you how Chrome handles a particular SCTP edge case, in complete, authoritative-sounding sentences, entirely from memory. Sometimes it's right. Sometimes the section number is off by one, or the requirement is subtly inverted, or the browser behavior is what the model expects rather than what ships. In a protocol stack that has to interoperate with real browsers, that's how you get a bug that passes every unit test and fails against Firefox.
So the rule was absolute: whenever a change touched a spec requirement or an interop assumption,
the agent had to fetch the primary source โ curl the actual RFC text from rfc-editor.org, or
read the actual Pion / libwebrtc / Firefox source โ and quote that, not its own recollection.
Here's what that caught. We rewrote SCTP's FORWARD-TSN generation (more below), and the agent's first framing was "this is behavior-preserving." I made it fetch RFC 3758. Reading the actual text of ยง3.2 revealed that the original code had a latent conformance violation: the Stream-Sequence field "MUST NOT report TSN's corresponding to DATA chunks that are marked as unordered," and the old code reported them anyway (harmless only because the receiver happened to ignore them). Our rewrite, which only tracked ordered streams, was the conformant behavior. The change went from "a refactor we hope is equivalent" to "a fix for a spec violation, with the paragraph quoted in the PR." Same for the SACK-piggyback work, where the correct requirement lives in RFC 4960 ยง6.1, not the ยง6.2 the model first reached for. When you're asking a maintainer to trust a protocol change, "here is the RFC paragraph, verbatim" is worth ten paragraphs of confident explanation.
Knowing the Enemy: Profiling Pion and rustrtc
You can't beat what you don't understand, so I pointed the same tools at the competition. This turned
out to be one of the best uses of the AI: I'd perf/bpftrace Pion and
rustrtc (a from-scratch Rust WebRTC stack that advertises
~2.8ร webrtc-rs), capture what they did differently at the syscall and CPU level, and then hand the
agent their actual source code and ask "we do X and pay for it โ what do they do here, and why is it
cheaper?"
That comparison is where several of our best changes came from:
bpftraceon Pion and rustrtc showed they emitted far fewer SACK-carrying datagrams per byte than we did. Reading rustrtc's source (with the agent) showed why: it batch-drains its receive channel and piggybacks the SACK onto outgoing DATA instead of sending each SACK as its own encrypted datagram. That directly inspired our batch-drain work.perfnormalized per delivered byte (not per wall-clock second โ that's confounded when two stacks deliver different volumes in a fixed window) showed the honest gap to rustrtc was CPU-work-per-byte, not syscalls: ~1.6ร the instructions per byte, from allocations, copies and crypto. That reframed the whole roadmap away from "clever syscall tricks" and toward "stop copying and allocating on the hot path."- Reading Pion's allocator behavior made it clear that Go's per-thread allocator was a real part of its low memory footprint, and that our thread-per-connection reactor model was inflating RSS via glibc per-thread arenas โ which set up the reactor-pool and back-pressure work.
Two honest notes from that comparison, because a blog post that only reports wins is marketing, not engineering:
- rustrtc is still leaner than us. Its from-scratch, single-runtime, tight-buffer architecture uses a fraction of our memory (tens of MB vs hundreds) and is still a bit faster per byte. We closed the throughput gap from ~1.9ร to near parity in the bulk regime and cut the per-byte instruction gap from 1.74ร to ~1.31ร, but "we beat Pion" is the honest headline, not "we beat everything."
- Some of what looked like a Pion advantage was a benchmark artifact. Pion's original 570 Mbps was on faster 2021 hardware; on a level playing field (same box, same config, steady-state rather than ramp-diluted cumulative averages) the picture is very different, as the numbers below show.
The Work, PR by PR
Eighteen PRs landed across the two repos (webrtc-rs/webrtc, the runtime/driver, and
webrtc-rs/rtc, the sans-I/O protocol core). They group into five themes.
1. Kill the scheduler overhead โ webrtc#813
The latency problem perf surfaced. Two changes: a coalesced wake (replace a per-message
blocking channel send that woke the driver once per 1 KB message with an atomic flag that fires at
most one wakeup per batch โ the same trick Pion's awakeWriteLoop uses), and an opt-in dedicated
reactor thread that pins a connection's driver to one thread so the tokio scheduler stops migrating
it across workers on every wake (context-switches and CPU-migrations were exploding 50โ100ร at โฅ4
workers). Together: 178 โ ~300 Mbps at the default runtime, closing the Pion gap from ~2ร to
~1.2ร. This is the PR that turned "webrtc-rs is hopelessly behind" into "it's a race."
2. Batch-drain the receive path โ webrtc#815 + rtc#124
The change directly inspired by profiling rustrtc. The driver now burst-reads the UDP socket
(drains up to 64 already-ready datagrams per wakeup instead of one per select! iteration), and the
SCTP handler defers its transmit flush to one pass per event-loop iteration, so N ingested DATA
chunks produce one SACK instead of one per two. Neither does much alone; together they roughly
doubled throughput and flipped webrtc-rs from behind Pion to ahead of it. This is the change the
earlier A/B comment
documented.
3. Stop copying and allocating โ the rtc per-byte rounds (#104, #105, #106, #107, #108, #111, #113, #121, #125, #130)
The long tail, and where the "CPU-per-byte" insight from the rustrtc comparison paid off. Highlights:
- Zero-copy payloads on the SCTP send and receive hot paths โ the datachannel send path was
doing an
alloc + memcpyper fragment for a buffer it already owned; the receive path did two copies where one would do (#121, #106). - Hardware CRC-32C and in-place AEAD, and running DTLS AES-GCM on
ringwith ARMv8 AES/PMULL enabled for the RustCrypto path (#107, #125) โ crypto was ~4โ6% of per-byte CPU and inherent, so making it hardware-accelerated mattered. - Algorithmic fixes, which are the ones I trust most because they're not micro-tuning:
O(Nยฒ)โO(N) datachannel queues (#106); and the big one, FORWARD-TSN generation from O(rwnd) to
O(streams) (#113).
perfhad flaggedAssociation::poll_transmitat ~9% self-time; annotating the hot addresses showed it was almost entirely hashmap probes from a per-SACK linear rescan of the whole in-flight window โ with PR-SCTP (no-retransmit) that's the entire receive window, ~99 million probes to build a one-entry map on a 512 MB transfer. We replaced it with an incrementally maintained map: โ12.3% instructions, andpoll_transmitdropped from 8.9% to 0.7%. (Pion has the identical O(window) scan, so this is a place we're now ahead of it.) This is also the change the RFC-3758 primary-source rule turned into a conformance fix. - Marshalling without allocations for SACK / FORWARD-TSN control chunks (#130) and bulk-copy / preallocate on the RTCP/media/NACK serialization paths (#105, #108).
4. Bound the memory โ webrtc#817 + rtc#127, webrtc#819 + rtc#129
Profiling rustrtc and Pion made clear that our peak RSS scaled badly, for two reasons: an unbounded send path, and one reactor thread per connection (glibc caches a per-thread arena on each). So:
- Opt-in send back-pressure (#817/#127):
writable().await/try_send()on the data channel, so an application flooding a channel faster than the network drains it blocks (or fails fast) instead of growing memory without bound. In a naive-flood stress test this cut peak RSS by ~50โ78% depending on the configured limit, with the happy path unchanged. - A bounded, shared reactor pool plus a configurable SCTP receive window (#819/#129):
replace thread-per-connection with a pool of
Nreactor threads (defaultavailable_parallelism), so thread count โ and the RSS that rides on it โ stops scaling with connection count. This is a scale-dependent tradeoff, and I measured the crossover honestly: below ~cores connections it's a mild loss (correctly off by default), but at SFU scale (50โ100 connections) it wins both axes โ ~+21% throughput and ~โ28% RSS at 50 connections, with RSS that stops growing. Combined with a smaller receive window it's ~โ70% RSS versus the default at that scale.
5. Batch the syscalls, finally โ webrtc#820
The GSO idea we declined early, revived once the batch-drain work meant sends actually coalesced.
UDP GSO on send and GRO on receive via quinn-udp. The measurement story here is a good
capstone for the whole "measure everything" theme, because the sign of the result flips by regime:
on a CPU-bound 10-pair run it's a clean win (โ24% instructions, โ35% wall on large messages); on a
single latency-bound connection, send-side batching turns a smooth 1-in-1-out ping-pong into bursty
N-at-once and regressed. bpftrace root-caused it, and the fix was an adaptive threshold โ
only take the GSO sendmsg path when a run has โฅ16 datagrams to batch โ which makes it a win (or
neutral) across every regime we measured. No single benchmark would have told us that.
The Numbers Today
Fresh A/B, run for this post on the same box (AMD Ryzen 7 1700, 8c/16t, Linux, loopback), current
webrtc-rs master versus Pion v4.2.16, both driven through the same flow-control harness: 1 KB
messages, 512 KB / 1 MB buffered-amount watermarks, per-interval steady-state throughput (warmup
skipped โ no ramp-diluted cumulative averages). Two configurations: unordered / no-retransmit
(Pion's own data-channels-flow-control example ships exactly this) and ordered / reliable (the
config the earlier issue-101 comment used). N=10 is ten concurrent connections, aggregate throughput.
Is this apples-to-apples? I checked, from source rather than memory, because it's the first thing
that invalidates a benchmark. The application config is identical on both stacks (ordering,
reliability, watermarks, 1 KB messages). The parameter that actually caps loopback throughput โ the
SCTP receive window โ is 1 MB on both (INITIAL_RECV_BUF_SIZE in rtc-sctp; initialRecvBufSize
in Pion's sctp v1.10.3, the version v4.2.16 pins). The one webrtc-rs-specific knob I enable is the
opt-in dedicated reactor thread โ so I report webrtc-rs both ways (plain tokio default and
+dedicated reactor), because the difference is large and hiding it would be dishonest. Pion has no
equivalent knob; its Go runtime provides that scheduling for free, which is rather the point.
(poop needs kernel perf counters, so these were captured with perf_event_paranoid lowered; the
Pion harness does a non-trickle full-SDP handshake to match the webrtc-rs one exactly.)
Steady-state throughput (Mbps, median of runs; ratio vs Pion in parens)
| configuration | Pion v4.2.16 | webrtc-rs (plain default) | webrtc-rs (+dedicated reactor) |
|---|---|---|---|
| Unordered / no-rtx, N=1 | 392 | 259 (0.66ร) | 689 (1.76ร) |
| Unordered / no-rtx, N=10 | 1681 | 2863 (1.70ร) | 5453 (3.24ร) |
| Ordered / reliable, N=1 | 385 | 184 (0.48ร) | 575 (1.49ร) |
| Ordered / reliable, N=10 | 1848 | 4297 (2.33ร) | 5296 (2.87ร) |
Read that honestly: single-connection, webrtc-rs's plain default loses to Pion (0.48โ0.66ร) โ that regime is latency-bound (the CPU sits ~90% idle) and Pion's Go scheduler beats plain tokio on round-trip latency. Turn on the one-line dedicated reactor and webrtc-rs pulls ahead (1.49โ1.76ร). In multi-connection aggregate โ the regime that saturates cores โ webrtc-rs wins even at its plain default (1.70โ2.33ร) and by ~2.9โ3.2ร with the reactor, because there per-byte CPU efficiency, not scheduling, decides it.
The poop counter tables
poop runs each binary head-to-head over fixed work (16 MB warmup + 48 MB measured per
connection, setup/teardown included) and reports deltas with significance markers (โก = significant
improvement, ๐ฉ = significant regression, no marker = within noise). Benchmark 1 is Pion (baseline);
Benchmark 2 is webrtc-rs +dedicated reactor.
Unordered / no-retransmit, N=1 (64 MB fixed work):
| measurement | Pion v4.2.16 | webrtc-rs +reactor | ฮ (webrtc vs Pion) |
|---|---|---|---|
| wall_time | 1.52 s | 1.43 s | โก โ5.9% ยฑ 4.0% |
| peak_rss | 23.6 MB | 11.6 MB | โก โ50.9% ยฑ 2.8% |
| cpu_cycles | 11.0 G | 2.81 G | โก โ74.5% ยฑ 0.8% |
| instructions | 8.94 G | 3.69 G | โก โ58.7% ยฑ 0.5% |
| cache_references | 1.09 G | 405 M | โก โ62.7% ยฑ 1.2% |
| cache_misses | 269 M | 88.2 M | โก โ67.2% ยฑ 1.9% |
| branch_misses | 72.2 M | 17.8 M | โก โ75.4% ยฑ 1.3% |
Unordered / no-retransmit, N=10 (640 MB fixed work):
| measurement | Pion v4.2.16 | webrtc-rs +reactor | ฮ (webrtc vs Pion) |
|---|---|---|---|
| wall_time | 3.40 s | 1.74 s | โก โ48.8% ยฑ 2.0% |
| peak_rss | 136 MB | 111 MB | โก โ18.4% ยฑ 4.5% |
| cpu_cycles | 106 G | 31.7 G | โก โ70.1% ยฑ 0.6% |
| instructions | 79.0 G | 35.6 G | โก โ54.9% ยฑ 0.5% |
| cache_references | 7.51 G | 3.03 G | โก โ59.6% ยฑ 0.4% |
| cache_misses | 1.28 G | 410 M | โก โ68.0% ยฑ 0.7% |
| branch_misses | 370 M | 111 M | โก โ70.0% ยฑ 0.7% |
Ordered / reliable, N=1 (64 MB fixed work):
| measurement | Pion v4.2.16 | webrtc-rs +reactor | ฮ (webrtc vs Pion) |
|---|---|---|---|
| wall_time | 1.48 s | 1.97 s | ๐ฉ +33.2% ยฑ 21.3% |
| peak_rss | 23.6 MB | 21.5 MB | โ8.7% ยฑ 24.9% |
| cpu_cycles | 11.1 G | 2.88 G | โก โ73.9% ยฑ 1.0% |
| instructions | 8.87 G | 3.72 G | โก โ58.1% ยฑ 0.6% |
| cache_references | 1.08 G | 415 M | โก โ61.7% ยฑ 1.1% |
| cache_misses | 269 M | 86.8 M | โก โ67.7% ยฑ 2.4% |
| branch_misses | 72.2 M | 18.1 M | โก โ74.9% ยฑ 1.5% |
Ordered / reliable, N=10 (640 MB fixed work):
| measurement | Pion v4.2.16 | webrtc-rs +reactor | ฮ (webrtc vs Pion) |
|---|---|---|---|
| wall_time | 3.49 s | 3.23 s | โ7.5% ยฑ 16.1% |
| peak_rss | 106 MB | 191 MB | ๐ฉ +79.5% ยฑ 22.8% |
| cpu_cycles | 102 G | 31.3 G | โก โ69.4% ยฑ 0.7% |
| instructions | 71.5 G | 35.7 G | โก โ50.1% ยฑ 0.6% |
| cache_references | 7.59 G | 3.35 G | โก โ55.9% ยฑ 1.0% |
| cache_misses | 1.42 G | 523 M | โก โ63.1% ยฑ 1.6% |
| branch_misses | 416 M | 126 M | โก โ69.7% ยฑ 0.9% |
Three things to take from those tables, in order of how much I trust them:
- Per-byte CPU efficiency is a decisive, robust win for webrtc-rs. Every cell, every counter: โ50% to โ75% cycles, instructions, cache references, cache misses and branch misses versus Pion for the same bytes transferred, with tight error bars. This is the Rust-vs-Go per-byte story, and it's the part I'd stake the post on. For an SFU paying an energy bill, this is the number that matters.
wall_timeis the noisy one, and it's not throughput. It includes per-connection setup (DTLS handshake + reactor-thread spawn, which webrtc-rs pays more of) and, at N=1, it's latency-bound so scheduling jitter dominates โ the ordered N=1wall_time ๐ฉ +33%is driven by a few outlier runs up to 4.2 s, while that same cell's CPU counters are all clean โก wins. That's why the steady-state throughput table above (setup excluded, measured window only) is the honest throughput figure, and it shows webrtc-rs ahead everywhere except the plain-default single connection.- Memory is a split decision. webrtc-rs is leaner on the unordered path (peak_rss โก โ51% at N=1, โก โ18% at N=10) but heavier on ordered/reliable at scale (๐ฉ +79.5%, 191 MB vs 106 MB) โ that's the reliable retransmit buffer, and it's exactly the cost the configurable receive window and reactor pool from theme #4 exist to bound.
For completeness and honesty, the bulk-transfer / flood regime โ the one that most favors rustrtc โ puts webrtc-rs at roughly parity-to-ahead of Pion on throughput and near rustrtc, while rustrtc still wins decisively on memory (tens of MB vs our hundreds). We beat Pion; we've closed most of the gap to the leanest from-scratch Rust stack; we're not the lightest thing in the room yet. That's the true state of it.
What I Actually Learned About Building with an AI
If you take one thing from this, let it be the shape of the collaboration, because it's replicable.
The AI was extraordinary at the parts humans are slow at: reading an unfamiliar 100k-line codebase, holding the SCTP state machine in its head, generating a dozen plausible hypotheses for a hotspot, writing the patch and the tests and the PR description, and reading a competitor's source to explain what they do differently. It compressed weeks of work into days.
It was also, left unchecked, a confident fabricator โ of benchmark intuitions that were a decade out
of date, of RFC section numbers, of "this is obviously the bottleneck" claims that perf flatly
contradicted. The value wasn't the AI or the tools alone. It was wiring the tools in as the AI's
reality check:
poopas the acceptance test. No change merges until an A/B run shows a significant delta in the direction claimed. The tool's significance markers are doing real work here โ they're what stop "it felt faster" from becoming a commit.perfandbpftraceas the hypothesis oracle. Don't ask the AI why it's slow; make it attribute the cost to a symbol and a syscall count, then optimize that, then prove the symbol shrank.- Primary sources as the anti-hallucination rule. For anything a browser or an RFC has an opinion about, the model quotes the fetched text, never its memory.
- Multiple benchmark regimes as the anti-overfit rule. A win has to survive latency-bound, CPU-bound, and memory-bound framing, or it ships with an explicit note about which regime it's for.
Adversarial review had the same flavor: for each substantive change I had a second agent try to refute it โ find the correctness bug, the missing test, the regressed regime โ rather than confirm it. That's how the FORWARD-TSN clear-logic test and the send back-pressure close/drop liveness fix got found before a maintainer had to find them.
Huge thanks to the maintainer for the reviews, for merging eighteen of these, and for pushing back where it mattered (the send back-pressure API went through three redesigns before it was right). The result is that the boring, honest goal of issue #101 โ "why is our data channel slower than Pion's?" โ now has a boring, honest answer: it isn't.
The benchmark harnesses, the competitor comparisons, and the A/B scripts are all reproducible; if you want to check the numbers on your own hardware, that's the whole point of building it this way. The appendix below has the exact steps.
Appendix: Reproduce It Yourself
Everything above was measured on an AMD Ryzen 7 1700 (8c/16t), Linux, loopback. Here are the exact steps.
Prerequisites: a recent Rust toolchain, Go โฅ 1.24, and poop.
poop reads kernel perf counters, so lower the paranoia level first (throughput-only runs don't need
this โ only the poop counter tables do):
sudo sysctl -w kernel.perf_event_paranoid=1
webrtc-rs side โ current master, with the in-tree multi-pair harness:
git clone https://github.com/webrtc-rs/webrtc && cd webrtc
git submodule update --init --recursive
cargo build --release --example data-channels-flow-control-multi
# one connection, unordered/no-retransmit, dedicated reactor on:
FLOW_PAIRS=1 FLOW_WARMUP_MB=16 FLOW_STOP_MB=48 FLOW_ORDERED=0 FLOW_DEDICATED_REACTOR=1 \
./target/release/examples/data-channels-flow-control-multi
The harness prints a FINAL <Mbps> aggregate over <N> pairs โฆ line โ that's per-interval
steady-state throughput (the FLOW_WARMUP_MB ramp is skipped, then throughput is measured over
FLOW_STOP_MB). Env knobs:
| variable | meaning | default |
|---|---|---|
FLOW_PAIRS |
concurrent connection pairs (aggregate) | 10 |
FLOW_WARMUP_MB |
ramp skipped before measuring | 64 |
FLOW_STOP_MB |
bytes measured per pair | 128 |
FLOW_ORDERED |
1 = ordered/reliable, else unordered/no-rtx |
0 |
FLOW_DEDICATED_REACTOR |
1 = opt-in dedicated reactor thread per PC |
1 |
Pion side โ pin the exact release, drop in a matching harness, build it:
git clone https://github.com/pion/webrtc pion-webrtc && cd pion-webrtc
git checkout v4.2.16
mkdir -p examples/flow-control-multi
# save the Go file below as examples/flow-control-multi/main.go
go build -o /tmp/pion-flow-multi ./examples/flow-control-multi
The Pion harness mirrors the webrtc-rs one exactly โ same env knobs, same 1 KB / 512 KBโ1 MB
watermarks, same per-interval steady-state measurement, and crucially a non-trickle handshake
(wait for gathering to complete, then exchange the full SDP) so it doesn't race on
AddICECandidate under concurrency:
// examples/flow-control-multi/main.go (SPDX-License-Identifier: MIT)
package main
import (
"fmt"
"os"
"strconv"
"sync"
"sync/atomic"
"time"
"github.com/pion/webrtc/v4"
)
const (
bufferedAmountLowThreshold uint64 = 512 * 1024
maxBufferedAmount uint64 = 1024 * 1024
msgSize = 1024
)
func check(err error) {
if err != nil {
panic(err)
}
}
func envInt(key string, def int) int {
if v := os.Getenv(key); v != "" {
if n, err := strconv.Atoi(v); err == nil {
return n
}
}
return def
}
func runPair(warmupBytes, stopBytes uint64, ordered bool, results chan<- float64) {
cfg := webrtc.Configuration{ICEServers: []webrtc.ICEServer{}}
offerPC, err := webrtc.NewPeerConnection(cfg)
check(err)
answerPC, err := webrtc.NewPeerConnection(cfg)
check(err)
var stop atomic.Bool
// Receiver: per-interval steady-state.
answerPC.OnDataChannel(func(dc *webrtc.DataChannel) {
var got, measureStartBytes uint64
var measuring atomic.Bool
var measureStart time.Time
dc.OnMessage(func(m webrtc.DataChannelMessage) {
got += uint64(len(m.Data))
if !measuring.Load() && got >= warmupBytes {
measureStart, measureStartBytes = time.Now(), got
measuring.Store(true)
}
if measuring.Load() && got-measureStartBytes >= stopBytes {
secs := time.Since(measureStart).Seconds()
mbps := float64((got-measureStartBytes)*8) / secs / (1024.0 * 1024.0)
if stop.CompareAndSwap(false, true) {
results <- mbps
}
}
})
})
// Sender.
maxRetransmits := uint16(0)
orderedFlag := ordered
opts := &webrtc.DataChannelInit{Ordered: &orderedFlag, MaxRetransmits: &maxRetransmits}
if ordered {
opts = &webrtc.DataChannelInit{Ordered: &orderedFlag}
}
dc, err := offerPC.CreateDataChannel("data", opts)
check(err)
dc.SetBufferedAmountLowThreshold(bufferedAmountLowThreshold)
sendMore := make(chan struct{}, 1)
dc.OnBufferedAmountLow(func() {
select {
case sendMore <- struct{}{}:
default:
}
})
dc.OnOpen(func() {
buf := make([]byte, msgSize)
for {
if stop.Load() {
return
}
if err := dc.Send(buf); err != nil {
return
}
if dc.BufferedAmount() > maxBufferedAmount {
select {
case <-sendMore:
case <-time.After(50 * time.Millisecond):
}
}
}
})
// Non-trickle handshake: wait for gathering, exchange full SDP.
offer, err := offerPC.CreateOffer(nil)
check(err)
offerGathered := webrtc.GatheringCompletePromise(offerPC)
check(offerPC.SetLocalDescription(offer))
<-offerGathered
check(answerPC.SetRemoteDescription(*offerPC.LocalDescription()))
answer, err := answerPC.CreateAnswer(nil)
check(err)
answerGathered := webrtc.GatheringCompletePromise(answerPC)
check(answerPC.SetLocalDescription(answer))
<-answerGathered
check(offerPC.SetRemoteDescription(*answerPC.LocalDescription()))
}
func main() {
pairs := envInt("FLOW_PAIRS", 10)
warmupBytes := uint64(envInt("FLOW_WARMUP_MB", 64)) * 1024 * 1024
stopBytes := uint64(envInt("FLOW_STOP_MB", 128)) * 1024 * 1024
ordered := os.Getenv("FLOW_ORDERED") == "1"
results := make(chan float64, pairs)
start := time.Now()
var wg sync.WaitGroup
for i := 0; i < pairs; i++ {
wg.Add(1)
go func() { defer wg.Done(); runPair(warmupBytes, stopBytes, ordered, results) }()
}
agg := 0.0
for i := 0; i < pairs; i++ {
agg += <-results
}
fmt.Printf("FINAL %.3f Mbps aggregate over %d pairs (ordered=%v) in %.3fs\n",
agg, pairs, ordered, time.Since(start).Seconds())
os.Exit(0)
}
The A/B itself. Set matching env for both binaries, read the FINAL lines for steady-state
throughput, and hand both to poop for the counter table (Benchmark 1 is the baseline):
export LC_ALL=C \
FLOW_PAIRS=10 FLOW_WARMUP_MB=16 FLOW_STOP_MB=48 FLOW_ORDERED=0 FLOW_DEDICATED_REACTOR=1
# steady-state throughput (run each a few times, take the median):
/tmp/pion-flow-multi
./target/release/examples/data-channels-flow-control-multi
# full hardware-counter A/B (pion = baseline, webrtc-rs = benchmark 2):
poop /tmp/pion-flow-multi ./target/release/examples/data-channels-flow-control-multi
Two gotchas that cost me time: export LC_ALL=C if your shell is in a comma-decimal locale (it
corrupts awk arithmetic in any wrapper script), and remember that poop's wall_time includes
per-connection setup and is latency-bound at N=1 โ the FINAL steady-state line is the honest
throughput number, the poop table is for the per-byte CPU and RSS counters.
Links
- The issue that started it: https://github.com/webrtc-rs/webrtc/issues/101
- webrtc (async): https://github.com/webrtc-rs/webrtc
- rtc (Sans-I/O core): https://github.com/webrtc-rs/rtc
- Stefano on GitHub: https://github.com/StefanoD
- poop: https://github.com/andrewrk/poop
- Discord: https://discord.gg/4Ju8UHdXMs
- Discussions: https://github.com/webrtc-rs/webrtc/discussions