orangefs: posix acl fix...

Al Viro pointed out that I broke some acl functionality...

 * ACLs could not be fully removed
 * posix_acl_chmod would be called while the old ACL was still cached
 * new mode propagated to orangefs server before ACL.

... when I tried to make sure that modes that got changed as a
result of ACL-sets would be sent back to the orangefs server.

Not wanting to try and change the code without having some cases to
test it with, I began to hunt for setfacl examples that were expressible
in pure mode. Along the way I found examples like the following
which confused me:

  user A had a file (/home/A/asdf) with mode 740
  user B was in user A's group
  user C was not in user A's group

  setfacl -m u:C:rwx /home/A/asdf

  The above setfacl caused ls -l /home/A/asdf to show a mode of 770,
  making it appear that all users in user A's group now had full access
  to /home/A/asdf, however, user B still only had read acces. Madness.

Anywho, I finally found that the above (whacky as it is) appears to
be "posixly on purpose" and explained in acl(5):

  If the ACL has an ACL_MASK entry, the group permissions correspond
  to the permissions of the ACL_MASK entry.
Signed-off-by: Mike Marshall <hubcap@omnibond.com>

fs/orangefs/acl.c

@@ -122,6 +122,8 @@ int orangefs_set_acl(struct inode *inode, struct posix_acl *acl, int type)
 	struct iattr iattr;
 	int rc;
 
+	memset(&iattr, 0, sizeof iattr);
+
 	if (type == ACL_TYPE_ACCESS && acl) {
 		/*
 		 * posix_acl_update_mode checks to see if the permissions
@@ -138,18 +140,17 @@ int orangefs_set_acl(struct inode *inode, struct posix_acl *acl, int type)
 			return error;
 		}
 
-		if (acl) {
-			rc = __orangefs_set_acl(inode, acl, type);
-		} else {
+		if (inode->i_mode != iattr.ia_mode)
 			iattr.ia_valid = ATTR_MODE;
-			rc = __orangefs_setattr(inode, &iattr);
-		}
 
-		return rc;
-
-	} else {
-		return -EINVAL;
 	}
+
+	rc = __orangefs_set_acl(inode, acl, type);
+
+	if (!rc && (iattr.ia_valid == ATTR_MODE))
+		rc = __orangefs_setattr(inode, &iattr);
+
+	return rc;
 }
 
 int orangefs_init_acl(struct inode *inode, struct inode *dir)