Merge branch 'caw-spam-and-captcha-docs' into 'master'

Add spam protection and CAPTCHA developer guide to docs

See merge request gitlab-org/gitlab!68434

doc/development/index.md

@@ -343,6 +343,7 @@ See [database guidelines](database/index.md).
 - [Dashboards for stage groups](stage_group_dashboards.md)
 - [Preventing transient bugs](transient/prevention-patterns.md)
 - [GitLab Application SLIs](application_slis/index.md)
+- [Spam protection and CAPTCHA development guide](spam_protection_and_captcha/index.md)
 
 ## Other GitLab Development Kit (GDK) guides
 

doc/development/spam_protection_and_captcha/graphql_api.md

+---
+stage: Manage
+group: Authentication and Authorization
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# GraphQL API spam protection and CAPTCHA support
+
+If the model can be modified via the GraphQL API, you must also add support to all of the
+relevant GraphQL mutations which may modify spammable or spam-related attributes. This
+definitely includes the `Create` and `Update` mutations, but may also include others, such as those
+related to changing a model's confidential/public flag.
+
+## Add support to the GraphQL mutations
+
+This implementation is very similar to the controller implementation. You create a `spam_params`
+instance based on the request, and pass it to the relevant Service class constructor.
+
+The three main differences from the controller implementation are:
+
+1. Use `include Mutations::SpamProtection` instead of `...JsonFormatActionsSupport`.
+1. Obtain the request from the context via `context[:request]` when creating the `SpamParams`
+   instance.
+1. After you create or updated the `Spammable` model instance, call `#check_spam_action_response!`
+   and pass it the model instance. This call will:
+    1. Perform the necessary spam checks on the model.
+    1. If spam is detected:
+       - Raise a `GraphQL::ExecutionError` exception.
+       - Include the relevant information added as error fields to the response via the `extensions:` parameter.
+       For more details on these fields, refer to the section on
+       [Spam and CAPTCHA support in the GraphQL API](../../api/graphql/index.md#resolve-mutations-detected-as-spam).
+
+       NOTE:
+       If you use the standard ApolloLink or Axios interceptor CAPTCHA support described
+       above, the field details are unimportant. They become important if you
+       attempt to use the GraphQL API directly to process a failed check for potential spam, and
+       resubmit the request with a solved CAPTCHA response.
+
+For example:
+
+```ruby
+module Mutations
+  module Widgets
+    class Create < BaseMutation
+      include Mutations::SpamProtection
+
+      def resolve(args)
+        spam_params = ::Spam::SpamParams.new_from_request(request: context[:request])
+
+        service_response = ::Widgets::CreateService.new(
+          project: project,
+          current_user: current_user,
+          params: args,
+          spam_params: spam_params
+        ).execute
+
+        widget = service_response.payload[:widget]
+        check_spam_action_response!(widget)
+
+        # If possible spam wasdetected, an exception would have been thrown by
+        # `#check_spam_action_response!`, so the normal resolve return logic can follow below.
+      end
+    end
+  end
+end
+```

doc/development/spam_protection_and_captcha/index.md

+---
+stage: Manage
+group: Authentication and Authorization
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Spam protection and CAPTCHA
+
+This guide provides an overview of how to add spam protection and CAPTCHA support to new areas of the
+GitLab application.
+
+## Add spam protection and CAPTCHA support to a new area
+
+To add this support, you must implement the following areas as applicable:
+
+1. [Model and Services](model_and_services.md): The basic prerequisite
+   changes to the backend code which are required to add spam or CAPTCHA API and UI support
+   for a feature which does not yet have support.
+1. REST API (Supported, documentation coming soon): The changes needed to add
+   spam or CAPTCHA support to Grape REST API endpoints. Refer to the related
+   [REST API documentation](../../api/index.md#resolve-requests-detected-as-spam).
+1. [GraphQL API](graphql_api.md): The changes needed to add spam or CAPTCHA support to GraphQL
+   mutations. Refer to the related
+   [GraphQL API documentation](../../api/graphql/index.md#resolve-mutations-detected-as-spam).
+1. [Web UI](web_ui.md): The various possible scenarios encountered when adding
+   spam/CAPTCHA support to the web UI, depending on whether the UI is JavaScript API-based (Vue or
+   plain JavaScript) or HTML-form (HAML) based.
+
+You should also perform manual exploratory testing of the new feature. Refer to
+[Exploratory testing](exploratory_testing.md) for more information.
+
+## Spam-related model and API fields
+
+Multiple levels of spam flagging determine how spam is handled. These levels are referenced in
+[`Spam::SpamConstants`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/services/spam/spam_constants.rb#L4-4),
+and used various places in the application, such as
+[`Spam::SpamActionService#perform_spam_service_check`](https://gitlab.com/gitlab-org/gitlab/blob/d7585b56c9e7dc69414af306d82906e28befe7da/app/services/spam/spam_action_service.rb#L61-61).
+
+The possible values include:
+
+- `BLOCK_USER`
+- `DISALLOW`
+- `CONDITIONAL_ALLOW`
+- `OVERRIDE_VIA_ALLOW_POSSIBLE_SPAM`
+- `ALLOW`
+- `NOOP`
+
+## Related topics
+
+- [Spam and CAPTCHA support in the GraphQL API](../../api/graphql/index.md#resolve-mutations-detected-as-spam)
+- [Spam and CAPTCHA support in the REST API](../../api/index.md#resolve-requests-detected-as-spam)
+- [reCAPTCHA Spam and Anti-bot Protection](../../integration/recaptcha.md)
+- [Akismet and Spam Logs](../../integration/akismet.md)

doc/development/spam_protection_and_captcha/model_and_services.md

+---
+stage: Manage
+group: Authentication and Authorization
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Model and services spam protection and CAPTCHA support
+
+Before adding any spam or CAPTCHA support to the REST API, GraphQL API, or Web UI, you must
+first add the necessary support to:
+
+1. The backend ActiveRecord models.
+1. The services layer.
+
+All or most of the following changes are required, regardless of the type of spam or CAPTCHA request
+implementation you are supporting. Some newer features which are completely based on the GraphQL API
+may not have any controllers, and don't require you to add the `mark_as_spam` action to the controller.
+
+To do this:
+
+1. [Add `Spammable` support to the ActiveRecord model](#add-spammable-support-to-the-activerecord-model).
+1. [Add support for the `mark_as_spam` action to the controller](#add-support-for-the-mark_as_spam-action-to-the-controller).
+1. [Add a call to SpamActionService to the execute method of services](#add-a-call-to-spamactionservice-to-the-execute-method-of-services).
+
+## Add `Spammable` support to the ActiveRecord model
+
+1. Include the `Spammable` module in the model class:
+
+   ```ruby
+   include Spammable
+   ```
+
+1. Add: `attr_spammable` to indicate which fields can be checked for spam. Up to
+   two fields per model are supported: a "`title`" and a "`description`". You can
+   designate which fields to consider the "`title`" or "`description`". For example,
+   this line designates the `content` field as the `description`:
+
+    ```ruby
+    attr_spammable :content, spam_description: true
+    ```
+
+1. Add a `#check_for_spam?` method implementation:
+
+   ```ruby
+   def check_for_spam?(user:)
+     # Return a boolean result based on various applicable checks, which may include
+     # which attributes have changed, the type of user, whether the data is publicly
+     # visible, and other criteria. This may vary based on the type of model, and
+     # may change over time as spam checking requirements evolve.
+   end
+   ```
+
+   Refer to other existing `Spammable` models'
+   implementations of this method for examples of the required logic checks.
+
+## Add support for the `mark_as_spam` action to the controller
+
+The `SpammableActions::AkismetMarkAsSpamAction` module adds support for a `#mark_as_spam` action
+to a controller. This controller allows administrators to manage spam for the associated
+`Spammable` model in the [Spam Log section](../../integration/akismet.md) of the Admin Area page.
+
+1. Include the `SpammableActions::AkismetMarkAsSpamAction` module in the controller.
+
+   ```ruby
+   include SpammableActions::AkismetMarkAsSpamAction
+   ```
+
+1. Add a `#spammable_path` method implementation. The spam administration page redirects
+   to this page after edits. Refer to other existing controllers' implementations
+   of this method for examples of the type of path logic required. In general, it should
+   be the `#show` action for the `Spammable` model's controller.
+
+   ```ruby
+   def spammable_path
+     widget_path(widget)
+   end
+   ```
+
+NOTE:
+There may be other changes needed to controllers, depending on how the feature is
+implemented. See [Web UI](web_ui.md) for more details.
+
+## Add a call to SpamActionService to the execute method of services
+
+This approach applies to any service which can persist spammable attributes:
+
+1. In the relevant Create or Update service under `app/services`, pass in a populated
+   `Spam::SpamParams` instance. (Refer to instructions later on in this page.)
+1. Use it and the `Spammable` model instance to execute a `Spam::SpamActionService` instance.
+1. If the spam check fails:
+   - An error is added to the model, which causes it to be invalid and prevents it from being saved.
+   - The `needs_recaptcha` property is set to `true`.
+
+   These changes to the model enable it for handling by the subsequent backend and frontend CAPTCHA logic.
+
+Make these changes to each relevant service:
+
+1. Change the constructor to take a `spam_params:` argument as a required named argument.
+
+   Using named arguments for the constructor helps you identify all the calls to
+   the constructor that need changing. It's less risky because the interpreter raises
+   type errors unless the caller is changed to pass the `spam_params` argument.
+   If you use an IDE (such as RubyMine) which supports this, your
+   IDE flags it as an error in the editor.
+
+1. In the constructor, set the `@spam_params` instance variable from the `spam_params` constructor
+   argument. Add an `attr_reader: :spam_params` in the `private` section of the class.
+
+1. In the `execute` method, add a call to execute the `Spam::SpamActionService`.
+   (You can also use `before_create` or `before_update`, if the service
+   uses that pattern.) This method uses named arguments, so its usage is clear if
+   you refer to existing examples. However, two important considerations exist:
+   1. The `SpamActionService` must be executed _after_ all necessary changes are made to
+      the unsaved (and dirty) `Spammable` model instance. This ordering ensures
+      spammable attributes exist to be spam-checked.
+   1. The `SpamActionService` must be executed _before_ the model is checked for errors and
+      attempting a `save`. If potential spam is detected in the model's changed attributes, we must prevent a save.
+
+```ruby
+module Widget
+  class CreateService < ::Widget::BaseService
+    # NOTE: We require the spam_params and do not default it to nil, because
+    # spam_checking is likely to be necessary.  However, if there is not a request available in scope
+    # in the caller (for example, a note created via email) and the required arguments to the
+    # SpamParams constructor are not otherwise available, spam_params: must be explicitly passed as nil.
+    def initialize(project:, current_user: nil, params: {}, spam_params:)
+      super(project: project, current_user: current_user, params: params)
+
+      @spam_params = spam_params
+    end
+
+    def execute
+      widget = Widget::BuildService.new(project, current_user, params).execute
+
+      # More code that may manipulate dirty model before it is spam checked.
+
+      # NOTE: do this AFTER the spammable model is instantiated, but BEFORE
+      # it is validated or saved.
+      Spam::SpamActionService.new(
+        spammable: widget,
+        spam_params: spam_params,
+        user: current_user,
+        # Or `action: :update` for a UpdateService or service for an existing model.
+        action: :create
+      ).execute
+
+      # Possibly more code related to saving model, but should not change any attributes.
+
+      widget.save
+    end
+
+    private
+
+    attr_reader :spam_params
+```

doc/development/spam_protection_and_captcha/web_ui.md

+---
+stage: Manage
+group: Authentication and Authorization
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Web UI spam protection and CAPTCHA support
+
+The approach for adding spam protection and CAPTCHA support to a new UI area of the GitLab application
+depends upon how the existing code is implemented.
+
+## Supported scenarios of request submissions
+
+Three different scenarios are supported. Two are used with JavaScript XHR/Fetch requests
+for either Apollo or Axios, and one is used only with standard HTML form requests:
+
+1. A JavaScript-based submission (possibly via Vue)
+    1. Using Apollo (GraphQL API via Fetch/XHR request)
+    1. Using Axios (REST API via Fetch/XHR request)
+1. A standard HTML form submission (HTML request)
+
+Some parts of the implementation depend upon which of these scenarios you must support.
+
+## Implementation tasks specific to JavaScript XHR/Fetch requests
+
+Two approaches are fully supported:
+
+1. Apollo, using the GraphQL API.
+1. Axios, using either the GraphQL API.
+
+The spam and CAPTCHA-related data communication between the frontend and backend requires no
+additional fields being added to the models. Instead, communication is handled:
+
+- Through custom header values in the request.
+- Through top-level JSON fields in the response.
+
+The spam and CAPTCHA-related logic is also cleanly abstracted into reusable modules and helper methods
+which can wrap existing logic, and only alter the existing flow if potential spam
+is detected or a CAPTCHA display is needed. This approach allows the spam and CAPTCHA
+support to be easily added to new areas of the application with minimal changes to
+existing logic. In the case of the frontend, potentially **zero** changes are needed!
+
+On the frontend, this is handled abstractly and transparently using `ApolloLink` for Apollo, and an
+Axios interceptor for Axios. The CAPTCHA display is handled by a standard GitLab UI / Pajamas modal
+component. You can find all the relevant frontend code under `app/assets/javascripts/captcha`.
+
+However, even though the actual handling of the request interception and
+modal is transparent, without any mandatory changes to the involved JavaScript or Vue components
+for the form or page, changes in request or error handling may be required. Changes are needed
+because the existing behavior may not work correctly: for example, if a failed or cancelled
+CAPTCHA display interrupts the normal request flow or UI updates.
+Careful exploratory testing of all scenarios is important to uncover any potential
+problems.
+
+This sequence diagram illustrates the normal CAPTCHA flow for JavaScript XHR/Fetch requests
+on the frontend:
+
+```mermaid
+sequenceDiagram
+    participant U as User
+    participant V as Vue/JS Application
+    participant A as ApolloLink or Axios Interceptor
+    participant G as GitLab API
+    U->>V: Save model
+    V->>A: Request
+    A->>G: Request
+    G--xA: Response with error and spam/CAPTCHA related fields
+    A->>U: CAPTCHA presented in modal
+    U->>A: CAPTCHA solved to obtain valid CAPTCHA response
+    A->>G: Request with valid CAPTCHA response and SpamLog ID in headers
+    G-->>A: Response with success
+    A-->>V: Response with success
+```
+
+The backend is also cleanly abstracted via mixin modules and helper methods. The three main
+changes required to the relevant backend controller actions (normally just `create`/`update`) are:
+
+1. Create a `SpamParams` parameter object instance based on the request, using the simple static
+   `#new_from_request` factory method. This method takes a request, and returns a `SpamParams` instance.
+1. Pass the created `SpamParams` instance as the `spam_params` named argument to the
+   Service class constructor, which you should have already added. If the spam check indicates
+   the changes to the model are possibly spam, then:
+   - An error is added to the model.
+   - The `needs_recaptcha` property on the model is set to true.
+1. Wrap the existing controller action return value (rendering or redirecting) in a block passed to
+   a `#with_captcha_check_json_format` helper method, which transparently handles:
+    1. Check if CAPTCHA is enabled, and if so, proceeding with the next step.
+    1. Checking if there the model contains an error, and the `needs_recaptcha` flag is true.
+       - If yes: Add the appropriate spam or CAPTCHA fields to the JSON response, and return
+         a `409 - Conflict` HTTP status code.
+       - If no (if CAPTCHA is disabled or if no spam was detected): The normal request return
+         logic passed in the block is run.
+
+Thanks to the abstractions, it's more straightforward to implement than it is to explain it.
+You don't have to worry much about the hidden details!
+
+Make these changes:
+
+## Add support to the controller actions
+
+If the feature's frontend submits directly to controller actions, and does not only use the GraphQL
+API, then you must add support to the appropriate controllers.
+
+The action methods may be directly in the controller class, or they may be abstracted
+to a module included in the controller class. Our example uses a module. The
+only difference when directly modifying the controller:
+`extend ActiveSupport::Concern` is not required.
+
+```ruby
+module WidgetsActions
+  # NOTE: This `extend` probably already exists, but it MUST be moved to occur BEFORE all
+  # `include` statements. Otherwise, confusing bugs may occur in which the methods
+  # in the included modules cannot be found.
+  extend ActiveSupport::Concern
+
+  include SpammableActions::CaptchaCheck::JsonFormatActionsSupport
+
+  def create
+    spam_params = ::Spam::SpamParams.new_from_request(request: request)
+    widget = ::Widgets::CreateService.new(
+      project: project,
+      current_user: current_user,
+      params: params,
+      spam_params: spam_params
+    ).execute
+
+    respond_to do |format|
+      format.json do
+        with_captcha_check_json_format do
+          # The action's existing `render json: ...` (or wrapper method) and related logic. Possibly
+          # including different rendering cases if the model is valid or not. It's all wrapped here
+          # within the `with_captcha_check_json_format` block. For example:
+          if widget.valid?
+            render json: serializer.represent(widget)
+          else
+            render json: { errors: widget.errors.full_messages }, status: :unprocessable_entity
+          end
+        end
+      end
+    end
+  end
+end
+```
+
+## Implementation tasks specific to HTML form requests
+
+Some areas of the application have not been converted to use the GraphQL API via
+a JavaScript client, but instead rely on standard Rails HAML form submissions via an
+`HTML` MIME type request. In these areas, the action returns a pre-rendered HTML (HAML) page
+as the response body. Unfortunately, in this case
+[it is not possible](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66427#note_636989204)
+to use any of the JavaScript-based frontend support as described above. Instead we must use an
+alternate approach which handles the rendering of the CAPTCHA form via a HAML template.
+
+Everything is still cleanly abstracted, and the implementation in the backend
+controllers is virtually identical to the JavaScript/JSON based approach. Replace the
+word `JSON` with `HTML` (using the appropriate case) in the module names and helper methods.
+
+The action methods might be directly in the controller, or they
+might be in a module. In this example, they are directly in the
+controller, and we also do an `update` method instead of `create`:
+
+```ruby
+class WidgetsController < ApplicationController
+  include SpammableActions::CaptchaCheck::HtmlFormatActionsSupport
+
+  def update
+    # Existing logic to find the `widget` model instance...
+
+    spam_params = ::Spam::SpamParams.new_from_request(request: request)
+    ::Widgets::UpdateService.new(
+      project: project,
+      current_user: current_user,
+      params: params,
+      spam_params: spam_params
+    ).execute(widget)
+
+    respond_to do |format|
+      format.html do
+        if widget.valid?
+          # NOTE: `spammable_path` is required by the `SpammableActions::AkismetMarkAsSpamAction`
+          # module, and it should have already been implemented on this controller according to
+          # the instructions above. It is reused here to avoid duplicating the route helper call.
+          redirect_to spammable_path
+        else
+          # If we got here, there were errors on the model instance - from a failed spam check
+          # and/or other validation errors on the model. Either way, we'll re-render the form,
+          # and if a CAPTCHA render is necessary, it will be automatically handled by
+          # `with_captcha_check_html_format`
+          with_captcha_check_html_format { render :edit }
+        end
+      end
+    end
+  end
+end
+```