The current public Jetstream instance has been running with `zstd` compression support for ~14 hours now, checking in on performance. The instance supports ~12 consumers right now with varying filters, but that fluctuates throughout the day. All consumers are currently caught up with live.

Curious, if you compared zstd against other options? Go's native zstd is "famous" to be less optimal, comparing to its C implementation. Could S2 (aka "better snappy") provide different trade-off for resource usage/speed?

Weismann Score?

Looking at a CPU profile, we can see half the total CPU (~0.18 cores) is being consumed by the ZSTD encoding process which encodes each message once for storage/broadcast and the Relay firehose consumer. Serving to subscribers is using less than 0.09 CPU cores, most of that time being syscalls

So the CPU usage for the compression is acceptable now? What did you tweak there in general to make it not-insane (ELI12 :)?

Great work!