The `x` button on labels incorrectly shows even for unauthorized
users. When unauthorized users click this the label is not removed
since there are backend authorization checks but this commit
fixes this UX inconsistency.