Fix performance problem of tags query

Closes gitlab-com/infrastructure#4550

See merge request gitlab-org/gitlab-ce!20555