• Ong Boon Leong's avatar
    net: stmmac: Enable RX via AF_XDP zero-copy · bba2556e
    Ong Boon Leong authored
    This patch adds the support for receiving packet via AF_XDP zero-copy
    mechanism.
    
    XDP ZC uses 1:1 mapping of XDP buffer to receive packet, therefore the
    use of split header is not used currently. The 'xdp_buff' is declared as
    union together with a struct that contains 'page', 'addr' and
    'page_offset' that are associated with primary buffer.
    
    RX buffers are now allocated either via page_pool or xsk pool. For RX
    buffers from xsk_pool they are allocated and deallocated using below
    functions:
    
     * stmmac_alloc_rx_buffers_zc(struct stmmac_priv *priv, u32 queue)
     * dma_free_rx_xskbufs(struct stmmac_priv *priv, u32 queue)
    
    With above functions now available, we then extend the following driver
    functions to support XDP ZC:
     * stmmac_reinit_rx_buffers()
     * __init_dma_rx_desc_rings()
     * init_dma_rx_desc_rings()
     * __free_dma_rx_desc_resources()
    
    Note: stmmac_alloc_rx_buffers_zc() may return -ENOMEM due to RX XDP
    buffer pool is not allocated (e.g. samples/bpf/xdpsock TX-only). But,
    it is still ok to let TX XDP ZC to continue, therefore, the -ENOMEM
    is silently ignored to let the driver succcessfully transition to XDP
    ZC mode for the said RX and TX queue.
    
    As XDP ZC buffer size is different, the DMA buffer size is required
    to be reprogrammed accordingly for RX DMA/Queue that is populated with
    XDP buffer from XSK pool.
    
    Next, to add or remove per-queue XSK pool, stmmac_xdp_setup_pool()
    will call stmmac_xdp_enable_pool() or stmmac_xdp_disable_pool()
    that in-turn coordinates the tearing down and setting up RX ring via
    RX buffers and descriptors removal and reallocation through
    stmmac_disable_rx_queue() and stmmac_enable_rx_queue(). In addition,
    stmmac_xsk_wakeup() is added to initiate XDP RX buffer replenishing
    by signalling user application to add available XDP frames back to
    FILL queue.
    
    For RX processing using XDP zero-copy buffer, stmmac_rx_zc() is
    introduced which is implemented with the assumption that RX split
    header is disabled. For XDP verdict is XDP_PASS, the XDP buffer is
    copied into a sk_buff allocated through stmmac_construct_skb_zc()
    and sent to Linux network GRO inside stmmac_dispatch_skb_zc(). Free RX
    buffers are then replenished using stmmac_rx_refill_zc()
    
    v2: introduce __stmmac_disable_all_queues() to contain the original code
        that does napi_disable() and then make stmmac_setup_tc_block_cb()
        to use it. Move synchronize_rcu() into stmmac_disable_all_queues()
        that eventually calls __stmmac_disable_all_queues(). Then,
        make both stmmac_release() and stmmac_suspend() to use
        stmmac_disable_all_queues(). Thanks David Miller for spotting the
        synchronize_rcu() issue in v1 patch.
    Signed-off-by: default avatarOng Boon Leong <boon.leong.ong@intel.com>
    Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
    bba2556e
stmmac_main.c 190 KB