1. 31 Dec, 2013 38 commits
  2. 13 Dec, 2013 2 commits
    • Josh Durgin's avatar
      libceph: resend all writes after the osdmap loses the full flag · 9a1ea2db
      Josh Durgin authored
      With the current full handling, there is a race between osds and
      clients getting the first map marked full. If the osd wins, it will
      return -ENOSPC to any writes, but the client may already have writes
      in flight. This results in the client getting the error and
      propagating it up the stack. For rbd, the block layer turns this into
      EIO, which can cause corruption in filesystems above it.
      
      To avoid this race, osds are being changed to drop writes that came
      from clients with an osdmap older than the last osdmap marked full.
      In order for this to work, clients must resend all writes after they
      encounter a full -> not full transition in the osdmap. osds will wait
      for an updated map instead of processing a request from a client with
      a newer map, so resent writes will not be dropped by the osd unless
      there is another not full -> full transition.
      
      This approach requires both osds and clients to be fixed to avoid the
      race. Old clients talking to osds with this fix may hang instead of
      returning EIO and potentially corrupting an fs. New clients talking to
      old osds have the same behavior as before if they encounter this race.
      
      Fixes: http://tracker.ceph.com/issues/6938Reviewed-by: default avatarSage Weil <sage@inktank.com>
      Signed-off-by: default avatarJosh Durgin <josh.durgin@inktank.com>
      9a1ea2db
    • Josh Durgin's avatar
      libceph: block I/O when PAUSE or FULL osd map flags are set · d29adb34
      Josh Durgin authored
      The PAUSEWR and PAUSERD flags are meant to stop the cluster from
      processing writes and reads, respectively. The FULL flag is set when
      the cluster determines that it is out of space, and will no longer
      process writes.  PAUSEWR and PAUSERD are purely client-side settings
      already implemented in userspace clients. The osd does nothing special
      with these flags.
      
      When the FULL flag is set, however, the osd responds to all writes
      with -ENOSPC. For cephfs, this makes sense, but for rbd the block
      layer translates this into EIO.  If a cluster goes from full to
      non-full quickly, a filesystem on top of rbd will not behave well,
      since some writes succeed while others get EIO.
      
      Fix this by blocking any writes when the FULL flag is set in the osd
      client. This is the same strategy used by userspace, so apply it by
      default.  A follow-on patch makes this configurable.
      
      __map_request() is called to re-target osd requests in case the
      available osds changed.  Add a paused field to a ceph_osd_request, and
      set it whenever an appropriate osd map flag is set.  Avoid queueing
      paused requests in __map_request(), but force them to be resent if
      they become unpaused.
      
      Also subscribe to the next osd map from the monitor if any of these
      flags are set, so paused requests can be unblocked as soon as
      possible.
      
      Fixes: http://tracker.ceph.com/issues/6079Reviewed-by: default avatarSage Weil <sage@inktank.com>
      Signed-off-by: default avatarJosh Durgin <josh.durgin@inktank.com>
      d29adb34