• Eric Dumazet's avatar
    net: use a per task frag allocator · 5640f768
    Eric Dumazet authored
    We currently use a per socket order-0 page cache for tcp_sendmsg()
    operations.
    
    This page is used to build fragments for skbs.
    
    Its done to increase probability of coalescing small write() into
    single segments in skbs still in write queue (not yet sent)
    
    But it wastes a lot of memory for applications handling many mostly
    idle sockets, since each socket holds one page in sk->sk_sndmsg_page
    
    Its also quite inefficient to build TSO 64KB packets, because we need
    about 16 pages per skb on arches where PAGE_SIZE = 4096, so we hit
    page allocator more than wanted.
    
    This patch adds a per task frag allocator and uses bigger pages,
    if available. An automatic fallback is done in case of memory pressure.
    
    (up to 32768 bytes per frag, thats order-3 pages on x86)
    
    This increases TCP stream performance by 20% on loopback device,
    but also benefits on other network devices, since 8x less frags are
    mapped on transmit and unmapped on tx completion. Alexander Duyck
    mentioned a probable performance win on systems with IOMMU enabled.
    
    Its possible some SG enabled hardware cant cope with bigger fragments,
    but their ndo_start_xmit() should already handle this, splitting a
    fragment in sub fragments, since some arches have PAGE_SIZE=65536
    
    Successfully tested on various ethernet devices.
    (ixgbe, igb, bnx2x, tg3, mellanox mlx4)
    Signed-off-by: default avatarEric Dumazet <edumazet@google.com>
    Cc: Ben Hutchings <bhutchings@solarflare.com>
    Cc: Vijay Subramanian <subramanian.vijay@gmail.com>
    Cc: Alexander Duyck <alexander.h.duyck@intel.com>
    Tested-by: default avatarVijay Subramanian <subramanian.vijay@gmail.com>
    Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
    5640f768
tcp.c 93.8 KB