Stop excess logs from invite email when group no longer exists

Closes #184

See merge request gitlab-org/security/gitlab!703