Merge branch 'docs-add-browserker-auth-documentation' into 'master'

Docs add browserker auth documentation

See merge request gitlab-org/gitlab!60452

doc/user/application_security/dast/browser_based.md

@@ -38,7 +38,8 @@ An example configuration might look like the following:
 include:
   - template: DAST.gitlab-ci.yml
 
-variables:
+dast:
+  variables:
     DAST_WEBSITE: "https://example.com"
     DAST_BROWSER_SCAN: "true"
 ```
@@ -64,8 +65,12 @@ The browser-based crawler can be configured using CI/CD variables.
 | `DAST_PASSWORD`                      | string          | `p@55w0rd`                        | The password to enter into the password field on the sign-in HTML form. |
 | `DAST_USERNAME_FIELD`                | selector        | `id:user`                         | A selector describing the username field on the sign-in HTML form. |
 | `DAST_PASSWORD_FIELD`                | selector        | `css:.password-field`             | A selector describing the password field on the sign-in HTML form. | 
-| `DAST_SUBMIT_FIELD`                  | selector        | `xpath://input[@value='Login']`   | A selector describing the element that when clicked submits the login form or the password form of a multi-page login process. |
+| `DAST_SUBMIT_FIELD`                  | selector        | `xpath://input[@value='Login']`   | A selector describing the element that when clicked submits the login form, or the password form of a multi-page login process. |
 | `DAST_FIRST_SUBMIT_FIELD`            | selector        | `.submit`                         | A selector describing the element that when clicked submits the username form of a multi-page login process. |
+| `DAST_BROWSER_AUTH_REPORT`           | boolean         | `true`                            | Used in combination with exporting the `gl-dast-debug-auth-report.html` artifact to aid in debugging authentication issues. |
+| `DAST_AUTH_VERIFICATION_URL`                | URL      | `http://site.com/welcome`         | Verifies successful authentication by checking the URL once the login form has been submitted. |
+| `DAST_BROWSER_AUTH_VERIFICATION_SELECTOR`   | selector | `css:.user-photo`                | Verifies successful authentication by checking for presence of a selector once the login form has been submitted. |
+| `DAST_BROWSER_AUTH_VERIFICATION_LOGIN_FORM` | boolean  | `true`                            | Verifies successful authentication by checking for the lack of a login form once the login form has been submitted. |
 
 The [DAST variables](index.md#available-variables) `SECURE_ANALYZERS_PREFIX`, `DAST_FULL_SCAN_ENABLED`, `DAST_AUTO_UPDATE_ADDONS`, `DAST_EXCLUDE_RULES`, `DAST_REQUEST_HEADERS`, `DAST_HTML_REPORT`, `DAST_MARKDOWN_REPORT`, `DAST_XML_REPORT`,
 `DAST_INCLUDE_ALPHA_VULNERABILITIES`, `DAST_PATHS_FILE`, `DAST_PATHS`, `DAST_ZAP_CLI_OPTIONS`, and `DAST_ZAP_LOG_CONFIGURATION` are also compatible with browser-based crawler scans.   
@@ -76,13 +81,47 @@ Selectors are used by CI/CD variables to specify the location of an element disp
 Selectors have the format `type`:`search string`. The crawler will search for the selector using the search string based on the type.
 
 | Selector type | Example                            | Description |
-| ------------- | ------------------------------ | ----------- |
+| ------------- | ---------------------------------- | ----------- |
 | `css`         | `css:.password-field`              | Searches for a HTML element having the supplied CSS selector. Selectors should be as specific as possible for performance reasons. |
 | `id`          | `id:element`                       | Searches for an HTML element with the provided element ID. |
 | `name`        | `name:element`                     | Searches for an HTML element with the provided element name. |
-| `xpath`       | `xpath://*[@id="my-button"]/a` | Searches for a HTML element with the provided XPath. Note that XPath searches are expected to be less performant than other searches. |
+| `xpath`       | `xpath://input[@id="my-button"]/a` | Searches for a HTML element with the provided XPath. Note that XPath searches are expected to be less performant than other searches. |
 | None provided | `a.click-me`                       | Defaults to searching using a CSS selector. |
 
+##### Find selectors with Google Chrome
+
+Chrome DevTools element selector tool is an effective way to find a selector. 
+
+1. Open Chrome and navigate to the page where you would like to find a selector, for example, the login page for your site.
+1. Open the `Elements` tab in Chrome DevTools with the keyboard shortcut `Command + Shift + c` in macOS or `Ctrl + Shift + c` in Windows.
+1. Select the `Select an element in the page to select it` tool.
+   ![search-elements](img/dast_auth_browser_scan_search_elements.png)
+1. Select the field on your page that you would like to know the selector for.
+1. Once the tool is active, highlight a field you wish to view the details of.
+   ![highlight](img/dast_auth_browser_scan_highlight.png)
+1. Once highlighted, you can see the element's details, including attributes that would make a good candidate for a selector.
+
+In this example, the `id="user_login"` appears to be a good candidate. You can use this as a selector as the DAST username field by setting `DAST_USERNAME_FIELD: "css:[id=user_login]"`, or more simply, `DAST_USERNAME_FIELD: "id:user_login"`.
+
+##### Choose the right selector
+
+Judicious choice of selector leads to a scan that is resilient to the application changing. 
+
+In order of preference, it is recommended to choose as selectors:
+
+- `id` fields. These are generally unique on a page, and rarely change.
+- `name` fields. These are generally unique on a page, and rarely change.
+- `class` values specific to the field, such as the selector `"css:.username"` for the `username` class on the username field.   
+- Presence of field specific data attributes, such as the selector, `"css:[data-username]"` when the `data-username` field has any value on the username field.
+- Multiple `class` hierarchy values, such as the selector `"css:.login-form .username"` when there are multiple elements with class `username` but only one nested inside the element with the class `login-form`.   
+
+When using selectors to locate specific fields we recommend you avoid searching on:
+
+- Any `id`, `name`, `attribute`, `class` or `value` that is dynamically generated.
+- Generic class names, such as `column-10` and `dark-grey`.
+- XPath searches as they are less performant than other selector searches.
+- Unscoped searches, such as those beginning with `css:*` and `xpath://*`.
+
 ## Vulnerability detection
 
 While the browser-based crawler crawls modern web applications efficiently, vulnerability detection is still managed by the standard DAST/Zed Attack Proxy (ZAP) solution.
@@ -93,6 +132,151 @@ When running a full scan, active vulnerability checks executed by DAST/ZAP do no
 For example, for a target website that contains forms with Anti-CSRF tokens, a passive scan will scan as intended because the browser displays pages/forms as if a user is viewing the page.
 However, active vulnerability checks run in a full scan will not be able to submit forms containing Anti-CSRF tokens. In such cases we recommend you disable Anti-CSRF tokens when running a full scan.
 
+## Authentication
+
+If your application hosts content only available to logged in users then you will likely get much higher crawl coverage of your application by configuring authentication.
+We recommended you periodically confirm that authentication is still working as this tends to break over
+time due to changes to the application.
+
+Authentication supports single form logins, multi-step login forms, and authenticating to URLs outside of the configured target URL.
+
+An example configuration that authenticates a user might look like the following:
+
+```yaml
+include:
+  - template: DAST.gitlab-ci.yml
+
+dast:
+  variables:
+    DAST_WEBSITE: "https://example.com"
+    DAST_BROWSER_SCAN: "true"
+    DAST_AUTH_URL: "https://login.example.com/"
+    DAST_USERNAME: "admin"
+    DAST_PASSWORD: "P@55w0rd!"
+    DAST_USERNAME_FIELD: "name:username"
+    DAST_PASSWORD_FIELD: "name:password"
+    DAST_SUBMIT_FIELD: "css:button[type='submit']"
+```  
+
+### Log in using automatic detection of the login form
+
+By providing a `DAST_USERNAME`, `DAST_PASSWORD`, and `DAST_AUTH_URL`, the browser-based crawler will attempt to authenticate to the
+target application by locating the login form based on a determination about whether or not the form contains username or password fields.
+
+Automatic detection is "best-effort", and depending on the application being scanned may provide either a resilient login experience or one that fails to authenticate the user.
+
+Login process:
+
+1. The `DAST_AUTH_URL` is loaded into the browser, and any forms on the page are located.
+   1. If a form contains a username and password field, `DAST_USERNAME` and `DAST_PASSWORD` is inputted into the respective fields, the form submit button is clicked and the user is logged in.
+   1. If a form contains only a username field, it is assumed that the login form is multi-step.
+      1. The `DAST_USERNAME` is inputted into the username field and the form submit button is clicked. 
+      1. The subsequent pages loads where it is expected that a form exists and contains a password field. If found, `DAST_PASSWORD` is inputted, form submit button is clicked and the user is logged in. 
+
+### Log in using explicit selection of the login form
+
+By providing a `DAST_USERNAME_FIELD`, `DAST_PASSWORD_FIELD`, and `DAST_SUBMIT_FIELD`, in addition to the fields required for automatic login,
+the browser-based crawler will attempt to authenticate to the target application by locating the login form based on the selectors provided.
+Most applications will benefit from this approach to authentication.
+
+Login process:
+
+1. The `DAST_AUTH_URL` is loaded into the browser, and any forms on the page are located.
+   1. If the `DAST_FIRST_SUBMIT_FIELD` is not defined, then `DAST_USERNAME` is inputted into `DAST_USERNAME_FIELD`, `DAST_PASSWORD` is inputted into `DAST_PASSWORD_FIELD`, `DAST_SUBMIT_FIELD` is clicked and the user is logged in.
+   1. If the `DAST_FIRST_SUBMIT_FIELD` is defined, then it is assumed that the login form is multi-step.
+      1. The `DAST_USERNAME` is inputted into the `DAST_USERNAME_FIELD` field and the `DAST_FIRST_SUBMIT_FIELD` is clicked.
+      1. The subsequent pages loads where the `DAST_PASSWORD` is inputted into the `DAST_PASSWORD_FIELD` field, the `DAST_SUBMIT_FIELD` is clicked and the user is logged in.
+      
+### Verifying successful login
+
+Once the login form has been submitted, the browser-based crawler determines if the login was successful. Unsuccessful attempts at authentication cause the scan to halt.
+
+Following the submission of the login form, authentication is determined to be unsuccessful when:
+ 
+- A `400` or `500` series HTTP response status code is returned.
+- A new cookie/browser storage value determined to be sufficiently random has not been set.
+
+In addition to these checks, the user can configure their own verification checks.
+Each of the following checks can be used in conjunction with one another, if none are configured by default the presence of a login form is checked.
+
+#### Verifying based on the URL
+
+When `DAST_AUTH_VERIFICATION_URL` is configured, the URL displayed in the browser tab post login form submission is directly compared to the URL in the CI/CD variable.
+If these are not exactly the same, authentication is deemed to be unsuccessful.
+
+For example:
+
+```yaml
+include:
+  - template: DAST.gitlab-ci.yml
+
+dast:
+  variables:
+    DAST_WEBSITE: "https://example.com"
+    DAST_BROWSER_SCAN: "true"
+    ...
+    DAST_AUTH_VERIFICATION_URL: "https://example.com/user/welcome"
+```  
+
+#### Verify based on presence of an element
+
+When `DAST_BROWSER_AUTH_VERIFICATION_SELECTOR` is configured, the page displayed in the browser tab is searched for an element described by the selector in the CI/CD variable.
+If no element is found, authentication is deemed to be unsuccessful.
+
+For example:
+
+```yaml
+include:
+  - template: DAST.gitlab-ci.yml
+
+dast: 
+  variables:
+    DAST_WEBSITE: "https://example.com"
+    DAST_BROWSER_SCAN: "true"
+    ...
+    DAST_BROWSER_AUTH_VERIFICATION_SELECTOR: "css:.welcome-user"
+```  
+
+#### Verify based on presence of a login form
+
+When `DAST_BROWSER_AUTH_VERIFICATION_LOGIN_FORM` is configured, the page displayed in the browser tab is searched for a form that is detected to be a login form.
+If any such form is found, authentication is deemed to be unsuccessful.
+
+For example:
+
+```yaml
+include:
+  - template: DAST.gitlab-ci.yml
+
+dast:
+  variables:
+    DAST_WEBSITE: "https://example.com"
+    DAST_BROWSER_SCAN: "true"
+    ...
+    DAST_BROWSER_AUTH_VERIFICATION_LOGIN_FORM: "true"
+```  
+
+### Configure the authentication debug output
+
+It is often difficult to understand the cause of an authentication failure when running DAST in a CI/CD pipeline.
+To assist users in debugging authentication issues, a debug report can be generated and saved as a job artifact.
+This HTML report contains all steps the browser crawler took, along with HTTP requests and responses, the Document Object Model (DOM) and screenshots.
+
+![dast-auth-report](img/dast_auth_report.jpg)
+
+An example configuration where the authentication debug report is exported may look like the following:
+
+```yaml
+dast:
+  variables:
+    DAST_WEBSITE: "https://example.com"
+    DAST_BROWSER_SCAN: "true"
+    ...
+    DAST_BROWSER_AUTH_REPORT: "true"
+  artifacts:
+    paths: [gl-dast-debug-auth-report.html]
+```
+
 ## Managing scan time
 
 It is expected that running the browser-based crawler will result in better coverage for many web applications, when compared to the normal GitLab DAST solution.
@@ -116,7 +300,8 @@ For example, the following job definition enables the browsing module and the au
 include:
   - template: DAST.gitlab-ci.yml
 
-variables:
+dast:
+  variables:
     DAST_WEBSITE: "https://my.site.com"
     DAST_BROWSER_SCAN: "true"
     DAST_BROWSER_LOG: "brows:debug,auth:debug"