• Chuck Lever's avatar
    NFS: Add static NFS I/O tracepoints · 8224b273
    Chuck Lever authored
    Tools like tcpdump and rpcdebug can be very useful. But there are
    plenty of environments where they are difficult or impossible to
    use. For example, we've had customers report I/O failures during
    workloads so heavy that collecting network traffic or enabling
    RPC debugging are themselves onerous.
    
    The kernel's static tracepoints are lightweight (less likely to
    introduce timing changes) and efficient (the trace data is compact).
    They also work in scenarios where capturing network traffic is not
    possible due to lack of hardware support (some InfiniBand HCAs) or
    where data or network privacy is a concern.
    
    Introduce tracepoints that show when an NFS READ, WRITE, or COMMIT
    is initiated, and when it completes. Record the arguments and
    results of each operation, which are not shown by existing sunrpc
    module's tracepoints.
    
    For instance, the recorded offset and count can be used to match an
    "initiate" event to a "done" event. If an NFS READ result returns
    fewer bytes than requested or zero, seeing the EOF flag can be
    probative. Seeing an NFS4ERR_BAD_STATEID result is also indication
    of a particular class of problems. The timing information attached
    to each event record can often be useful as well.
    
    Usage example:
    
    [root@manet tmp]# trace-cmd record -e nfs:*initiate* -e nfs:*done
    /sys/kernel/debug/tracing/events/nfs/*initiate*/filter
    /sys/kernel/debug/tracing/events/nfs/*done/filter
    Hit Ctrl^C to stop recording
    ^CKernel buffer statistics:
      Note: "entries" are the entries left in the kernel ring buffer and are not
            recorded in the trace data. They should all be zero.
    
    CPU: 0
    entries: 0
    overrun: 0
    commit overrun: 0
    bytes: 3680
    oldest event ts:    78.367422
    now ts:   100.124419
    dropped events: 0
    read events: 74
    
    ... and so on.
    Signed-off-by: default avatarChuck Lever <chuck.lever@oracle.com>
    Signed-off-by: default avatarTrond Myklebust <trond.myklebust@primarydata.com>
    8224b273
nfstrace.h 23.8 KB