• Paolo Abeni's avatar
    udp: do fwd memory scheduling on dequeue · 7c13f97f
    Paolo Abeni authored
    A new argument is added to __skb_recv_datagram to provide
    an explicit skb destructor, invoked under the receive queue
    lock.
    The UDP protocol uses such argument to perform memory
    reclaiming on dequeue, so that the UDP protocol does not
    set anymore skb->desctructor.
    Instead explicit memory reclaiming is performed at close() time and
    when skbs are removed from the receive queue.
    The in kernel UDP protocol users now need to call a
    skb_recv_udp() variant instead of skb_recv_datagram() to
    properly perform memory accounting on dequeue.
    
    Overall, this allows acquiring only once the receive queue
    lock on dequeue.
    
    Tested using pktgen with random src port, 64 bytes packet,
    wire-speed on a 10G link as sender and udp_sink as the receiver,
    using an l4 tuple rxhash to stress the contention, and one or more
    udp_sink instances with reuseport.
    
    nr sinks	vanilla		patched
    1		440		560
    3		2150		2300
    6		3650		3800
    9		4450		4600
    12		6250		6450
    
    v1 -> v2:
     - do rmem and allocated memory scheduling under the receive lock
     - do bulk scheduling in first_packet_length() and in udp_destruct_sock()
     - avoid the typdef for the dequeue callback
    Suggested-by: default avatarEric Dumazet <edumazet@google.com>
    Acked-by: default avatarHannes Frederic Sowa <hannes@stressinduktion.org>
    Signed-off-by: default avatarPaolo Abeni <pabeni@redhat.com>
    Acked-by: default avatarEric Dumazet <edumazet@google.com>
    Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
    7c13f97f
udp.c 65.9 KB