• Eric Dumazet's avatar
    tcp: better TCP_SKB_CB layout to reduce cache line misses · 971f10ec
    Eric Dumazet authored
    TCP maintains lists of skb in write queue, and in receive queues
    (in order and out of order queues)
    
    Scanning these lists both in input and output path usually requires
    access to skb->next, TCP_SKB_CB(skb)->seq, and TCP_SKB_CB(skb)->end_seq
    
    These fields are currently in two different cache lines, meaning we
    waste lot of memory bandwidth when these queues are big and flows
    have either packet drops or packet reorders.
    
    We can move TCP_SKB_CB(skb)->header at the end of TCP_SKB_CB, because
    this header is not used in fast path. This allows TCP to search much faster
    in the skb lists.
    
    Even with regular flows, we save one cache line miss in fast path.
    
    Thanks to Christoph Paasch for noticing we need to cleanup
    skb->cb[] (IPCB/IP6CB) before entering IP stack in tx path,
    and that I forgot IPCB use in tcp_v4_hnd_req() and tcp_v4_save_options().
    Signed-off-by: default avatarEric Dumazet <edumazet@google.com>
    Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
    971f10ec
tcp_ipv6.c 50.1 KB