• Eric Dumazet's avatar
    net: busy-poll: allow preemption in sk_busy_loop() · 217f6974
    Eric Dumazet authored
    After commit 4cd13c21 ("softirq: Let ksoftirqd do its job"),
    sk_busy_loop() needs a bit of care :
    softirqs might be delayed since we do not allow preemption yet.
    
    This patch adds preemptiom points in sk_busy_loop(),
    and makes sure no unnecessary cache line dirtying
    or atomic operations are done while looping.
    
    A new flag is added into napi->state : NAPI_STATE_IN_BUSY_POLL
    
    This prevents napi_complete_done() from clearing NAPIF_STATE_SCHED,
    so that sk_busy_loop() does not have to grab it again.
    
    Similarly, netpoll_poll_lock() is done one time.
    
    This gives about 10 to 20 % improvement in various busy polling
    tests, especially when many threads are busy polling in
    configurations with large number of NIC queues.
    
    This should allow experimenting with bigger delays without
    hurting overall latencies.
    
    Tested:
     On a 40Gb mlx4 NIC, 32 RX/TX queues.
    
     echo 70 >/proc/sys/net/core/busy_read
     for i in `seq 1 40`; do echo -n $i: ; ./super_netperf $i -H lpaa24 -t UDP_RR -- -N -n; done
    
        Before:      After:
     1:   90072   92819
     2:  157289  184007
     3:  235772  213504
     4:  344074  357513
     5:  394755  458267
     6:  461151  487819
     7:  549116  625963
     8:  544423  716219
     9:  720460  738446
    10:  794686  837612
    11:  915998  923960
    12:  937507  925107
    13: 1019677  971506
    14: 1046831 1113650
    15: 1114154 1148902
    16: 1105221 1179263
    17: 1266552 1299585
    18: 1258454 1383817
    19: 1341453 1312194
    20: 1363557 1488487
    21: 1387979 1501004
    22: 1417552 1601683
    23: 1550049 1642002
    24: 1568876 1601915
    25: 1560239 1683607
    26: 1640207 1745211
    27: 1706540 1723574
    28: 1638518 1722036
    29: 1734309 1757447
    30: 1782007 1855436
    31: 1724806 1888539
    32: 1717716 1944297
    33: 1778716 1869118
    34: 1805738 1983466
    35: 1815694 2020758
    36: 1893059 2035632
    37: 1843406 2034653
    38: 1888830 2086580
    39: 1972827 2143567
    40: 1877729 2181851
    Signed-off-by: default avatarEric Dumazet <edumazet@google.com>
    Cc: Willem de Bruijn <willemb@google.com>
    Cc: Adam Belay <abelay@google.com>
    Cc: Tariq Toukan <tariqt@mellanox.com>
    Cc: Yuval Mintz <Yuval.Mintz@cavium.com>
    Cc: Ariel Elior <ariel.elior@cavium.com>
    Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
    217f6974
dev.c 209 KB