dropbear: Don't waste transfer time in favour of small-memory machines defaults
I recently discovered that dropbear, compared to openssh over same link
with same host and user keys, transfers files several times slower -
sometimes up to an order of magnitude slower. This equally applies to
both fast LAN IPv4 networks (tried on 1Gbps) and relatively slow "over
re6stnet" networks with geographically separated client and server.
The problem turned out to be dropbear has very small receive window size
default:
https://github.com/mkj/dropbear/blob/DROPBEAR_0.53.1/options.h#L258
---- 8< ----
/* Window size limits. These tend to be a trade-off between memory
usage and network performance: */
/* Size of the network receive window. This amount of memory is allocated
as a per-channel receive buffer. Increasing this value can make a
significant difference to network performance. 24kB was empirically
chosen for a 100mbit ethernet network. The value can be altered at
runtime with the -W argument. */
#ifndef DEFAULT_RECV_WINDOW
#define DEFAULT_RECV_WINDOW 24576
#endif
---- 8< ----
this is maybe appropriate for embedded systems, but small receive window
affects networking throughput a lot: e.g. I've tried to run several
file downloading benchmarks and changing receive window size from
default 24K to 1M (this is maximum according to dropbear documentation) on
client side made the downloading several times faster and comparable to
downloading speed with using openssh as client.
Dropbear has -W option for setting receive window size at runtime, so we could
go through all places which use dropbear and add appropriate `-W ...` there.
But I think fixing dropbear itself to have more sane defaults for
not-very-memory-constrained devices (= devices we run slapos on) is preferable
- this way all services will automatically benefit from transfer speedup
without taking action on their side with only doing recompilation/redeployment.
Besides changing only recv window size at runtime breaks compatibility with
openssh: if we only do `-W 1M` on server and try to upload data with openssh as
client, dropbear complains
[3302] Apr 17 23:10:06 Exit (slapuser2): Bad packet size 32777
and connection terminates. Thus RECV_MAX_PAYLOAD_LEN increase is also
required, which cannot be done via option at runtime:
https://github.com/mkj/dropbear/blob/DROPBEAR_0.53.1/options.h#L268
---- 8< ----
/* Maximum size of a received SSH data packet - this _MUST_ be >= 32768
in order to interoperate with other implementations */
#ifndef RECV_MAX_PAYLOAD_LEN
#define RECV_MAX_PAYLOAD_LEN 32768
#endif
---- 8< ----
So let's increase DEFAULT_RECV_WINDOW to 1M and RECV_MAX_PAYLOAD_LEN
appropriately (experimentally found that at 512K the complain goes
away).
Below is rough benchmark before and after the change (timings are not
very precise, because machines are otherwise busy and network
fluctuates, but I think they are reasonable in showing we have at least
several times improvement):
before after
download 1M (re6stnet, slow link) ~12s ~4s
upload 1M (re6stnet, slow link) ~12s ~4s
download 100M (1Gbps LAN, IPv4) ~12s ~4s
upload 100M (1Gbps LAN, IPv4) ~12s ~4s
legend:
client=dbclient
server=dropbear
NOTE the original problem equally applies to latest dropbear master, so it is
not because we are running old 0.53.1 ( from 2011 ! )
NOTE wrt transfer speed openssh is still faster - not an order of
magnitude, but somewhat. Also openssh uses much less CPU compared to
dropbear.
/cc @Tyagov, @klaus
@nexedi, why at all we use dropbear in the first place instead of openssh?
/reviewed-by @rafael
/reviewed-on !68
Showing
Please register or sign in to comment