Optimize query performance in specific case

This becomes especially bad in this query from /api/v4/projects
endpoint:

```
SELECT COUNT(*)
FROM
  (SELECT 1 AS one
   FROM "projects"
   WHERE (EXISTS
            (SELECT 1
             FROM "project_authorizations"
             WHERE "project_authorizations"."user_id" = 1
               AND (project_authorizations.project_id = projects.id)
            )
            OR projects.visibility_level IN (10,20)
          )
     AND "projects"."visibility_level" = 0
     AND "projects"."pending_delete" = FALSE
   LIMIT 10001) subquery_for_count
```

Note we attempt to limit the damage with `LIMIT 10001` but Postgres
keeps trying to find any row that will match the impossible visiblity
level combination of (10,20) & (0), which goes way beyond 10001 rows.

So we omit the OR in the case where we encounter a visibility level
set which is null.

app/finders/projects_finder.rb

@@ -82,7 +82,29 @@ class ProjectsFinder < UnionFinder
       if private_only?
         current_user.authorized_projects
       else
-        Project.public_or_visible_to_user(current_user)
+        default_visibility_levels = Gitlab::VisibilityLevel.levels_for_user(current_user)
+
+        # This is an optimization - surprisingly PostgreSQL does not optimize
+        # for this.
+        #
+        # If the default visiblity level and desired visiblity level filter
+        # cancels each other out, don't use the SQL clause for visibility level.
+        # In fact, this then becames equivalent to just authorized projects for
+        # the user.
+        #
+        # E.g.
+        # (EXISTS(<authorized_projects>) OR projects.visibility_level IN (10,20))
+        #   AND "projects"."visibility_level" = 0
+        #
+        # is essentially
+        # EXISTS(<authorized_projects>) AND "projects"."visibility_level" = 0
+        #
+        # See https://gitlab.com/gitlab-org/gitlab/issues/37007
+        if params[:visibility_level].present? && !default_visibility_levels.include?(params[:visibility_level])
+          current_user.authorized_projects
+        else
+          Project.public_or_visible_to_user(current_user)
+        end
       end
     end
   end

changelogs/unreleased/projects_finder_visibility_optimization.yml

+---
+title: Optimize query when Projects API requests private visibility level
+merge_request: 20594
+author:
+type: performance