Automatic merge of gitlab-org/gitlab master

changelogs/unreleased/jv-rack-attack-user-bypass.yml

+---
+title: Add user ID based allowlist for Rack::Attack
+merge_request: 49127
+author:
+type: changed

doc/development/agent/user_stories.md

+---
+stage: Configure
+group: Configure
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
+# Kubernetes Agent user stories **(PREMIUM ONLY)**
+
+The [personas in action](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#user-personas)
+for the Kubernetes Agent are:
+
+- [Sasha, the Software Developer](https://about.gitlab.com/handbook/marketing/strategic-marketing/roles-personas/#sasha-software-developer).
+- [Allison, the Application Operator](https://about.gitlab.com/handbook/marketing/strategic-marketing/roles-personas/#allison-application-ops).
+- [Priyanka, the Platform Engineer](https://about.gitlab.com/handbook/marketing/strategic-marketing/roles-personas/#priyanka-platform-engineer).
+
+[Devon, the DevOps engineer](https://about.gitlab.com/handbook/marketing/strategic-marketing/roles-personas/#devon-devops-engineer)
+is intentionally excluded here, as DevOps is more of a role than a persona.
+
+There are various workflows to support, so some user stories might seem to contradict each other. They don't.
+
+## Software Developer user stories
+
+<!-- vale gitlab.FirstPerson = NO -->
+
+- As a Software Developer, I want to push my code, and move to the next development task,
+  to work on business applications.
+- As a Software Developer, I want to set necessary dependencies and resource requirements
+  together with my application code, so my code runs fine after deployment.
+
+<!-- vale gitlab.FirstPerson = YES -->
+
+## Application Operator user stories
+
+<!-- vale gitlab.FirstPerson = NO -->
+
+- As an Application Operator, I want to standardize the deployments used by my teams,
+  so I can support all teams with minimal effort.
+- As an Application Operator, I want to have a single place to define all the deployments,
+  so I can assure security fixes are applied everywhere.
+- As an Application Operator, I want to offer a set of predefined templates to
+  Software Developers, so they can get started quickly and can deploy to production
+  without my intervention, and I am not a bottleneck.
+- As an Application Operator, I want to know exactly what changes are being deployed,
+  so I can fulfill my SLAs.
+- As an Application Operator, I want deep insights into what versions of my applications
+  are running and want to be able to debug them, so I can fix operational issues.
+- As an Application Operator, I want application code to be automatically deployed
+  to staging environments when new versions are available.
+- As an Application Operator, I want to follow my preferred deployment strategy,
+  so I can move code into production in a reliable way.
+- As an Application Operator, I want review all code before it's deployed into production,
+  so I can fulfill my SLAs.
+- As an Application Operator, I want to be notified before deployment when new code needs my attention,
+  so I can review it swiftly.
+
+<!-- vale gitlab.FirstPerson = YES -->
+
+## Platform Engineer user stories
+
+<!-- vale gitlab.FirstPerson = NO -->
+
+- As a Platform Engineer, I want to restrict customizations to preselected values
+  for Operators, so I can fulfill my SLAs.
+- As a Platform Engineer, I want to allow some level of customization to Operators,
+  so I don't become a bottleneck.
+- As a Platform Engineer, I want to define all deployments in a single place, so
+  I can assure security fixes are applied everywhere.
+- As a Platform Engineer, I want to define the infrastructure by code, so my
+  infrastructure management is testable, reproducible, traceable, and scalable.
+- As a Platform Engineer, I want to define various policies that applications must
+  follow, so that I can fulfill my SLAs.
+- As a Platform Engineer, I want approved tooling for log management and persistent storage,
+  so I can scale, secure, and manage them as needed.
+- As a Platform Engineer, I want to be alerted when my infrastructure differs from
+  its definition, so I can make sure that everything is configured as expected.
+
+<!-- vale gitlab.FirstPerson = YES -->

doc/user/admin_area/settings/user_and_ip_rate_limits.md

@@ -59,6 +59,29 @@ are marked with `"throttle_safelist":"throttle_bypass_header"` in
 To disable the bypass mechanism, make sure the environment variable
 `GITLAB_THROTTLE_BYPASS_HEADER` is unset or empty.
 
+## Allowing specific users to bypass authenticated request rate limiting
+
+Similarly to the bypass header described above, it is possible to allow
+a certain set of users to bypass the rate limiter. This only applies
+to authenticated requests: with unauthenticated requests, by definition
+GitLab does not know who the user is.
+
+The allowlist is configured as a comma-separated list of user IDs in
+the `GITLAB_THROTTLE_USER_ALLOWLIST` environment variable. If you want
+users 1, 53 and 217 to bypass the authenticated request rate limiter,
+the allowlist configuration would be `1,53,217`.
+
+- For [Omnibus](https://docs.gitlab.com/omnibus/settings/environment-variables.html),
+  set `'GITLAB_THROTTLE_USER_ALLOWLIST' => '1,53,217'` in `gitlab_rails['env']`.
+- For source installations, set `export GITLAB_THROTTLE_USER_ALLOWLIST=1,53,217`
+  in `/etc/default/gitlab`.
+
+Requests that bypassed the rate limiter because of the user allowlist
+are marked with `"throttle_safelist":"throttle_user_allowlist"` in
+[`production_json.log`](../../../administration/logs.md#production_jsonlog).
+
+At application startup, the allowlist is logged in [`auth.log`](../../../administration/logs.md#authlog).
+
 <!-- ## Troubleshooting
 
 Include any troubleshooting steps that you can foresee. If you know beforehand what issues

doc/user/clusters/agent/index.md

@@ -82,8 +82,8 @@ Upgrade your agent installations together with GitLab upgrades. To decide which
 1. Open the [GITLAB_KAS_VERSION](https://gitlab.com/gitlab-org/gitlab/-/blob/master/GITLAB_KAS_VERSION) file from the GitLab Repository, which contains the latest `agentk` version associated with the `master` branch.
 1. Change the `master` branch and select the Git tag associated with your version. For instance, you could change it to GitLab [v13.5.3-ee release](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.5.3-ee/GITLAB_KAS_VERSION)
 
-The available `agentk` versions can be found in
-[its container registry](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/container_registry/eyJuYW1lIjoiZ2l0bGFiLW9yZy9jbHVzdGVyLWludGVncmF0aW9uL2dpdGxhYi1hZ2VudC9hZ2VudGsiLCJ0YWdzX3BhdGgiOiIvZ2l0bGFiLW9yZy9jbHVzdGVyLWludGVncmF0aW9uL2dpdGxhYi1hZ2VudC9yZWdpc3RyeS9yZXBvc2l0b3J5LzEyMjMyMDUvdGFncz9mb3JtYXQ9anNvbiIsImlkIjoxMjIzMjA1LCJjbGVhbnVwX3BvbGljeV9zdGFydGVkX2F0IjpudWxsfQ==).
+The available `agentk` and `kas` versions can be found in
+[the container registry](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/container_registry/).
 
 ### Install the Kubernetes Agent Server
 
@@ -537,7 +537,7 @@ issue is in progress, directly edit the deployment with the
 `kubectl edit deployment gitlab-kas` command, and change `--listen-websocket=true` to `--listen-websocket=false`. After running that command, you should be able to use
 `grpc://gitlab-kas.<YOUR-NAMESPACE>:5005`.
 
-#### Agent logs - Decompressor is not installed for grpc-encoding
+### Agent logs - Decompressor is not installed for grpc-encoding
 
 ```plaintext
 {"level":"warn","time":"2020-11-05T05:25:46.916Z","msg":"GetConfiguration.Recv failed","error":"rpc error: code = Unimplemented desc = grpc: Decompressor is not installed for grpc-encoding \"gzip\""}

doc/user/infrastructure/index.md

@@ -31,6 +31,12 @@ Amazon S3 or Google Cloud Storage. Its features include:
 
 Read more on setting up and [using GitLab Managed Terraform states](terraform_state.md)
 
+WARNING:
+Like any other job artifact, Terraform plan data is [viewable by anyone with Guest access](../permissions.md) to the repository.
+Neither Terraform nor GitLab encrypts the plan file by default. If your Terraform plan
+includes sensitive data such as passwords, access tokens, or certificates, GitLab strongly
+recommends encrypting plan output or modifying the project visibility settings.
+
 ## Terraform integration in Merge Requests
 
 Collaborating around Infrastructure as Code (IaC) changes requires both code changes and expected infrastructure changes to be checked and approved. GitLab provides a solution to help collaboration around Terraform code changes and their expected effects using the Merge Request pages. This way users don't have to build custom tools or rely on 3rd party solutions to streamline their IaC workflows.

lib/gitlab/rack_attack.rb

@@ -11,6 +11,13 @@ module Gitlab
       Rack::Attack.throttled_response_retry_after_header = true
       # Configure the throttles
       configure_throttles(rack_attack)
+
+      configure_user_allowlist
+    end
+
+    def self.configure_user_allowlist
+      @user_allowlist = nil
+      user_allowlist
     end
 
     def self.configure_throttles(rack_attack)
@@ -25,7 +32,7 @@ module Gitlab
       throttle_or_track(rack_attack, 'throttle_authenticated_api', Gitlab::Throttle.authenticated_api_options) do |req|
         if req.api_request? &&
            Gitlab::Throttle.settings.throttle_authenticated_api_enabled
-          req.authenticated_user_id([:api])
+          req.throttled_user_id([:api])
         end
       end
 
@@ -41,7 +48,7 @@ module Gitlab
       throttle_or_track(rack_attack, 'throttle_authenticated_web', Gitlab::Throttle.authenticated_web_options) do |req|
         if req.web_request? &&
            Gitlab::Throttle.settings.throttle_authenticated_web_enabled
-          req.authenticated_user_id([:api, :rss, :ics])
+          req.throttled_user_id([:api, :rss, :ics])
         end
       end
 
@@ -60,7 +67,7 @@ module Gitlab
            req.api_request? &&
            req.protected_path? &&
            Gitlab::Throttle.protected_paths_enabled?
-          req.authenticated_user_id([:api])
+          req.throttled_user_id([:api])
         end
       end
 
@@ -69,7 +76,7 @@ module Gitlab
            req.web_request? &&
            req.protected_path? &&
            Gitlab::Throttle.protected_paths_enabled?
-          req.authenticated_user_id([:api, :rss, :ics])
+          req.throttled_user_id([:api, :rss, :ics])
         end
       end
 
@@ -95,6 +102,14 @@ module Gitlab
 
       dry_run_config.split(',').map(&:strip).include?(name)
     end
+
+    def self.user_allowlist
+      @user_allowlist ||= begin
+        list = UserAllowlist.new(ENV['GITLAB_THROTTLE_USER_ALLOWLIST'])
+        Gitlab::AuthLogger.info(gitlab_throttle_user_allowlist: list.to_a)
+        list
+      end
+    end
   end
 end
 ::Gitlab::RackAttack.prepend_if_ee('::EE::Gitlab::RackAttack')

lib/gitlab/rack_attack/request.rb

@@ -7,8 +7,15 @@ module Gitlab
         !(authenticated_user_id([:api, :rss, :ics]) || authenticated_runner_id)
       end
 
-      def authenticated_user_id(request_formats)
-        request_authenticator.user(request_formats)&.id
+      def throttled_user_id(request_formats)
+        user_id = authenticated_user_id(request_formats)
+
+        if Gitlab::RackAttack.user_allowlist.include?(user_id)
+          Gitlab::Instrumentation::Throttle.safelist = 'throttle_user_allowlist'
+          return
+        end
+
+        user_id
       end
 
       def authenticated_runner_id
@@ -49,6 +56,10 @@ module Gitlab
 
       private
 
+      def authenticated_user_id(request_formats)
+        request_authenticator.user(request_formats)&.id
+      end
+
       def request_authenticator
         @request_authenticator ||= Gitlab::Auth::RequestAuthenticator.new(self)
       end

lib/gitlab/rack_attack/user_allowlist.rb

+# frozen_string_literal: true
+
+require 'set'
+
+module Gitlab
+  module RackAttack
+    class UserAllowlist
+      extend Forwardable
+
+      def_delegators :@set, :empty?, :include?, :to_a
+
+      def initialize(list)
+        @set = Set.new
+
+        list.to_s.split(',').each do |id|
+          @set << Integer(id) unless id.blank?
+        rescue ArgumentError
+          Gitlab::AuthLogger.error(message: 'ignoring invalid user allowlist entry', entry: id)
+        end
+      end
+    end
+  end
+end

package.json

@@ -80,7 +80,7 @@
     "deckar01-task_list": "^2.3.1",
     "diff": "^3.4.0",
     "document-register-element": "1.14.3",
-    "dompurify": "^2.2.2",
+    "dompurify": "^2.2.3",
     "dropzone": "^4.2.0",
     "editorconfig": "^0.15.3",
     "emoji-regex": "^7.0.3",

spec/lib/gitlab/rack_attack/user_allowlist_spec.rb

+# frozen_string_literal: true
+
+require 'spec_helper'
+
+RSpec.describe Gitlab::RackAttack::UserAllowlist do
+  using RSpec::Parameterized::TableSyntax
+
+  subject { described_class.new(input)}
+
+  where(:input, :elements) do
+    nil | []
+    '' |  []
+    '123' | [123]
+    '123,456' | [123, 456]
+    '123,foobar, 456,' | [123, 456]
+  end
+
+  with_them do
+    it 'has the expected elements' do
+      expect(subject).to contain_exactly(*elements)
+    end
+
+    it 'implements empty?' do
+      expect(subject.empty?).to eq(elements.empty?)
+    end
+
+    it 'implements include?' do
+      unless elements.empty?
+        expect(subject).to include(elements.first)
+      end
+    end
+  end
+end

spec/lib/gitlab/rack_attack_spec.rb

@@ -75,5 +75,22 @@ RSpec.describe Gitlab::RackAttack, :aggregate_failures do
         expect(fake_rack_attack).to have_received(:throttle).with(throttle.to_s, throttles[throttle])
       end
     end
+
+    context 'user allowlist' do
+      subject { described_class.user_allowlist }
+
+      it 'is empty' do
+        described_class.configure(fake_rack_attack)
+
+        expect(subject).to be_empty
+      end
+
+      it 'reflects GITLAB_THROTTLE_USER_ALLOWLIST' do
+        stub_env('GITLAB_THROTTLE_USER_ALLOWLIST', '123,456')
+        described_class.configure(fake_rack_attack)
+
+        expect(subject).to contain_exactly(123, 456)
+      end
+    end
   end
 end

spec/support/shared_examples/requests/rack_attack_shared_examples.rb

@@ -23,6 +23,11 @@ RSpec.shared_examples 'rate-limited token-authenticated requests' do
     settings_to_set[:"#{throttle_setting_prefix}_period_in_seconds"] = period_in_seconds
   end
 
+  after do
+    stub_env('GITLAB_THROTTLE_USER_ALLOWLIST', nil)
+    Gitlab::RackAttack.configure_user_allowlist
+  end
+
   context 'when the throttle is enabled' do
     before do
       settings_to_set[:"#{throttle_setting_prefix}_enabled"] = true
@@ -30,6 +35,8 @@ RSpec.shared_examples 'rate-limited token-authenticated requests' do
     end
 
     it 'rejects requests over the rate limit' do
+      expect(Gitlab::Instrumentation::Throttle).not_to receive(:safelist=)
+
       # At first, allow requests under the rate limit.
       requests_per_period.times do
         make_request(request_args)
@@ -40,6 +47,18 @@ RSpec.shared_examples 'rate-limited token-authenticated requests' do
       expect_rejection { make_request(request_args) }
     end
 
+    it 'does not reject requests if the user is in the allowlist' do
+      stub_env('GITLAB_THROTTLE_USER_ALLOWLIST', user.id.to_s)
+      Gitlab::RackAttack.configure_user_allowlist
+
+      expect(Gitlab::Instrumentation::Throttle).to receive(:safelist=).with('throttle_user_allowlist').at_least(:once)
+
+      (requests_per_period + 1).times do
+        make_request(request_args)
+        expect(response).not_to have_gitlab_http_status(:too_many_requests)
+      end
+    end
+
     it 'allows requests after throttling and then waiting for the next period' do
       requests_per_period.times do
         make_request(request_args)
@@ -167,6 +186,11 @@ RSpec.shared_examples 'rate-limited web authenticated requests' do
     settings_to_set[:"#{throttle_setting_prefix}_period_in_seconds"] = period_in_seconds
   end
 
+  after do
+    stub_env('GITLAB_THROTTLE_USER_ALLOWLIST', nil)
+    Gitlab::RackAttack.configure_user_allowlist
+  end
+
   context 'when the throttle is enabled' do
     before do
       settings_to_set[:"#{throttle_setting_prefix}_enabled"] = true
@@ -174,6 +198,8 @@ RSpec.shared_examples 'rate-limited web authenticated requests' do
     end
 
     it 'rejects requests over the rate limit' do
+      expect(Gitlab::Instrumentation::Throttle).not_to receive(:safelist=)
+
       # At first, allow requests under the rate limit.
       requests_per_period.times do
         request_authenticated_web_url
@@ -184,6 +210,18 @@ RSpec.shared_examples 'rate-limited web authenticated requests' do
       expect_rejection { request_authenticated_web_url }
     end
 
+    it 'does not reject requests if the user is in the allowlist' do
+      stub_env('GITLAB_THROTTLE_USER_ALLOWLIST', user.id.to_s)
+      Gitlab::RackAttack.configure_user_allowlist
+
+      expect(Gitlab::Instrumentation::Throttle).to receive(:safelist=).with('throttle_user_allowlist').at_least(:once)
+
+      (requests_per_period + 1).times do
+        request_authenticated_web_url
+        expect(response).not_to have_gitlab_http_status(:too_many_requests)
+      end
+    end
+
     it 'allows requests after throttling and then waiting for the next period' do
       requests_per_period.times do
         request_authenticated_web_url

yarn.lock

@@ -4181,10 +4181,10 @@ domhandler@^2.3.0:
   dependencies:
     domelementtype "1"
 
-dompurify@^2.2.2:
-  version "2.2.2"
-  resolved "https://registry.yarnpkg.com/dompurify/-/dompurify-2.2.2.tgz#cb8c2b1a2f3c8a0b565127504ae4eedec176a972"
-  integrity sha512-BsGR4nDLaC5CNBnyT5I+d5pOeaoWvgVeg6Gq/aqmKYWMPR07131u60I80BvExLAJ0FQEIBQ1BTicw+C5+jOyrg==
+dompurify@^2.2.2, dompurify@^2.2.3:
+  version "2.2.3"
+  resolved "https://registry.yarnpkg.com/dompurify/-/dompurify-2.2.3.tgz#ec653ba521b39f397c2ca045769438d593ea8a9f"
+  integrity sha512-8Hv7Q0FuwD9rWoB6qI2eZsfKbGXfoUVuGHHrE15vgk4ReOKwOkSgbqb2OMFtc0d5besOEkoLkcyuV10zQ2X5gw==
 
 domutils@^1.5.1:
   version "1.6.2"