• Linas Vepstas's avatar
    spidernet: Cure RX ram full bug · 4c4bd5a9
    Linas Vepstas authored
    This patch fixes a rare deadlock that can occur when the kernel
    is not able to empty out the RX ring quickly enough. Below follows
    a detailed description of the bug and the fix.
    
    As long as the OS can empty out the RX buffers at a rate faster than
    the hardware can fill them, there is no problem. If, for some reason,
    the OS fails to empty the RX ring fast enough, the hardware GDACTDPA
    pointer will catch up to the head, notice the not-empty condition,
    ad stop. However, RX packets may still continue arriving on the wire.
    The spidernet chip can save some limited number of these in local RAM.
    When this local ram fills up, the spider chip will issue an interrupt
    indicating this (GHIINT0STS will show ERRINT, and the GRMFLLINT bit
    will be set in GHIINT1STS).  When te RX ram full condition occurs,
    a certain bug/feature is triggered that has to be specially handled.
    This section describes the special handling for this condition.
    
    When the OS finally has a chance to run, it will empty out the RX ring.
    In particular, it will clear the descriptor on which the hardware had
    stopped. However, once the hardware has decided that a certain
    descriptor is invalid, it will not restart at that descriptor; instead
    it will restart at the next descr. This potentially will lead to a
    deadlock condition, as the tail pointer will be pointing at this descr,
    which, from the OS point of view, is empty; the OS will be waiting for
    this descr to be filled. However, the hardware has skipped this descr,
    and is filling the next descrs. Since the OS doesn't see this, there
    is a potential deadlock, with the OS waiting for one descr to fill,
    while the hardware is waiting for a differen set of descrs to become
    empty.
    
    A call to show_rx_chain() at this point indicates the nature of the
    problem. A typical print when the network is hung shows the following:
    
    net eth1: Spider RX RAM full, incoming packets might be discarded!
    net eth1: Total number of descrs=256
    net eth1: Chain tail located at descr=255
    net eth1: Chain head is at 255
    net eth1: HW curr desc (GDACTDPA) is at 0
    net eth1: Have 1 descrs with stat=xa0800000
    net eth1: HW next desc (GDACNEXTDA) is at 1
    net eth1: Have 127 descrs with stat=x40800101
    net eth1: Have 1 descrs with stat=x40800001
    net eth1: Have 126 descrs with stat=x40800101
    net eth1: Last 1 descrs with stat=xa0800000
    
    Both the tail and head pointers are pointing at descr 255, which is
    marked xa... which is "empty". Thus, from the OS point of view, there
    is nothing to be done. In particular, there is the implicit assumption
    that everything in front of the "empty" descr must surely also be empty,
    as explained in the last section. The OS is waiting for descr 255 to
    become non-empty, which, in this case, will never happen.
    
    The HW pointer is at descr 0. This descr is marked 0x4.. or "full".
    Since its already full, the hardware can do nothing more, and thus has
    halted processing. Notice that descrs 0 through 254 are all marked
    "full", while descr 254 and 255 are empty. (The "Last 1 descrs" is
    descr 254, since tail was at 255.) Thus, the system is deadlocked,
    and there can be no forward progress; the OS thinks there's nothing
    to do, and the hardware has nowhere to put incoming data.
    
    This bug/feature is worked around with the spider_net_resync_head_ptr()
    routine. When the driver receives RX interrupts, but an examination
    of the RX chain seems to show it is empty, then it is probable that
    the hardware has skipped a descr or two (sometimes dozens under heavy
    network conditions). The spider_net_resync_head_ptr() subroutine will
    search the ring for the next full descr, and the driver will resume
    operations there.  Since this will leave "holes" in the ring, there
    is also a spider_net_resync_tail_ptr() that will skip over such holes.
    Signed-off-by: default avatarLinas Vepstas <linas@austin.ibm.com>
    Signed-off-by: default avatarJeff Garzik <jeff@garzik.org>
    4c4bd5a9
spider_net.c 68.1 KB