Merge branch '334578-document-new-variable-in-dast-api-docs' into 'master'

Document OVERRIDES_CMD_VERBOSE env var for API Fuzzing and DAST API

See merge request gitlab-org/gitlab!74556

doc/user/application_security/api_fuzzing/index.md

@@ -461,7 +461,7 @@ Follow these steps to provide the bearer token with `FUZZAPI_OVERRIDES_ENV`:
    ```
 
 1. To validate that authentication is working, run an API fuzzing test and review the fuzzing logs
-   and the test API's application logs.
+   and the test API's application logs. See the [overrides section](#overrides) for more information about override commands.
 
 ##### Token generated at test runtime
 
@@ -495,7 +495,7 @@ variables:
   FUZZAPI_PROFILE: Quick
   FUZZAPI_OPENAPI: test-api-specification.json
   FUZZAPI_TARGET_URL: http://test-deployment/
-  FUZZAPI_OVERRIDES_FILE: output/api-fuzzing-overrides.json
+  FUZZAPI_OVERRIDES_FILE: api-fuzzing-overrides.json
 ```
 
 To validate that authentication is working, run an API fuzzing test and review the fuzzing logs and
@@ -537,7 +537,7 @@ variables:
   FUZZAPI_PROFILE: Quick-10
   FUZZAPI_OPENAPI: test-api-specification.json
   FUZZAPI_TARGET_URL: http://test-deployment/
-  FUZZAPI_OVERRIDES_FILE: output/api-fuzzing-overrides.json
+  FUZZAPI_OVERRIDES_FILE: api-fuzzing-overrides.json
   FUZZAPI_OVERRIDES_CMD: renew_token.py
   FUZZAPI_OVERRIDES_INTERVAL: 300
 ```
@@ -577,6 +577,9 @@ profile increases as the number of tests increases.
 |[`FUZZAPI_OVERRIDES_FILE`](#overrides)                       | Path to a JSON file containing overrides. |
 |[`FUZZAPI_OVERRIDES_ENV`](#overrides)                        | JSON string containing headers to override. |
 |[`FUZZAPI_OVERRIDES_CMD`](#overrides)                        | Overrides command. |
+|[`FUZZAPI_OVERRIDES_CMD_VERBOSE`](#overrides)                | When set to any value. It shows overrides command output as part of the job output. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334578) in GitLab 14.8. |
+|`FUZZAPI_PRE_SCRIPT`                                         | Run user command or script before scan session starts. |
+|`FUZZAPI_POST_SCRIPT`                                        | Run user command or script after scan session has finished. |
 |[`FUZZAPI_OVERRIDES_INTERVAL`](#overrides)                   | How often to run overrides command in seconds. Defaults to `0` (once). |
 |[`FUZZAPI_HTTP_USERNAME`](#http-basic-authentication)        | Username for HTTP authentication. |
 |[`FUZZAPI_HTTP_PASSWORD`](#http-basic-authentication)        | Password for HTTP authentication. |
@@ -756,7 +759,7 @@ variables:
   FUZZAPI_PROFILE: Quick
   FUZZAPI_OPENAPI: test-api-specification.json
   FUZZAPI_TARGET_URL: http://test-deployment/
-  FUZZAPI_OVERRIDES_FILE: output/api-fuzzing-overrides.json
+  FUZZAPI_OVERRIDES_FILE: api-fuzzing-overrides.json
 ```
 
 #### Using a CI/CD variable
@@ -801,16 +804,181 @@ variables:
 
 If the value must be generated or regenerated on expiration, you can provide a program or script for
 the API fuzzer to execute on a specified interval. The provided script runs in an Alpine Linux
-container that has Python 3 and Bash installed. If the Python script requires additional packages,
-it must detect this and install the packages at runtime. The script creates the overrides JSON file
-as defined above.
+container that has Python 3 and Bash installed. 
+
+You have to set the environment variable `FUZZAPI_OVERRIDES_CMD` to the program or script you would like
+to execute. The provided command creates the overrides JSON file as defined previously.
+
+You might want to install other scripting runtimes like NodeJS or Ruby, or maybe you need to install a dependency for
+your overrides command. In this case, we recommend setting the `FUZZAPI_PRE_SCRIPT` to the file path of a script which
+provides those prerequisites. The script provided by `FUZZAPI_PRE_SCRIPT` is executed once, before the analyzer starts.
+
+See the [Alpine Linux package management](https://wiki.alpinelinux.org/wiki/Alpine_Linux_package_management) 
+page for information about installing Alpine Linux packages.
 
 You must provide three CI/CD variables, each set for correct operation:
 
 - `FUZZAPI_OVERRIDES_FILE`: File generated by the provided command.
-- `FUZZAPI_OVERRIDES_CMD`: Command to generate JSON file.
+- `FUZZAPI_OVERRIDES_CMD`: Overrides command in charge of generating the overrides JSON file periodically.
 - `FUZZAPI_OVERRIDES_INTERVAL`: Interval in seconds to run command.
 
+Optionally:
+
+- `FUZZAPI_PRE_SCRIPT`: Script to install runtimes or dependencies before the analyzer starts.
+
+```yaml
+stages:
+     - fuzz
+
+include:
+  - template: API-Fuzzing.gitlab-ci.yml
+
+variables:
+  FUZZAPI_PROFILE: Quick
+  FUZZAPI_OPENAPI: test-api-specification.json
+  FUZZAPI_TARGET_URL: http://test-deployment/
+  FUZZAPI_OVERRIDES_FILE: api-fuzzing-overrides.json
+  FUZZAPI_OVERRIDES_CMD: renew_token.py
+  FUZZAPI_OVERRIDES_INTERVAL: 300
+```
+
+#### Debugging overrides
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334578) in GitLab 14.8.
+
+By default the output of the overrides command is hidden. If the overrides command returns a non zero exit code, the command is displayed as part of your job output. Optionally, you can set the variable `FUZZAPI_OVERRIDES_CMD_VERBOSE` to any value in order to display overrides command output as it is generated. This is useful when testing your overrides script, but should be disabled afterwards as it slows down testing.
+
+It is also possible to write messages from your script to a log file that is collected when the job completes or fails. The log file must be created in a specific location and follow a naming convention.
+
+Adding some basic logging to your overrides script is useful in case the script fails unexpectedly during normal running of the job. The log file is automatically included as an artifact of the job, allowing you to download it after the job has finished.
+
+Following our example, we provided `renew_token.py` in the environmental variable `FUZZAPI_OVERRIDES_CMD`. Please notice two things in the script:
+
+- Log file is saved in the location indicated by the environment variable `CI_PROJECT_DIR`.
+- Log file name should match `gl-*.log`.
+
+```python
+#!/usr/bin/env python
+
+# Example of an overrides command
+
+# Override commands can update the overrides json file
+# with new values to be used.  This is a great way to
+# update an authentication token that will expire
+# during testing.
+
+import logging
+import json
+import os
+import requests
+import backoff
+
+# [1] Store log file in directory indicated by env var CI_PROJECT_DIR
+working_directory = os.environ['CI_PROJECT_DIR']
+
+# [2] File name should match the pattern: gl-*.log
+log_file_path = os.path.join(working_directory, 'gl-user-overrides.log')
+
+# Set up logger
+logging.basicConfig(filename=log_file_path, level=logging.DEBUG)
+
+# Use `backoff` decorator to retry in case of transient errors.
+@backoff.on_exception(backoff.expo,
+                      (requests.exceptions.Timeout,
+                       requests.exceptions.ConnectionError),
+                       max_time=30)
+def get_auth_response():
+    return requests.get('https://authorization.service/api/get_api_token', auth=(os.environ['AUTH_USER'], os.environ['AUTH_PWD']))
+
+
+# In our example, access token is retrieved from a given endpoint
+try:
+
+    # Performs a http request, response sample: 
+    # { "Token" : "b5638ae7-6e77-4585-b035-7d9de2e3f6b3" }
+    response = get_auth_response()
+
+    # Check that the request is successful. may raise `requests.exceptions.HTTPError`
+    response.raise_for_status()
+
+    # Gets JSON data
+    response_body = response.json()
+
+# If needed specific exceptions can be caught
+# requests.ConnectionError                  : A network connection error problem occurred
+# requests.HTTPError                        : HTTP request returned an unsuccessful status code. [Response.raise_for_status()]
+# requests.ConnectTimeout                   : The request timed out while trying to connect to the remote server
+# requests.ReadTimeout                      : The server did not send any data in the allotted amount of time.
+# requests.TooManyRedirects                 : The request exceeds the configured number of maximum redirections
+# requests.exceptions.RequestException      : All exceptions that related to Requests
+except requests.exceptions.RequestException as requests_error:
+    # logs  exceptions  related to `Requests`
+    logging.error(f'Error, failed while performing HTTP request. Error message: {requests_error}')
+    raise
+except requests.exceptions.JSONDecodeError as json_decode_error:
+    # logs errors related decoding JSON response
+    logging.error(f'Error, failed while decoding JSON response. Error message: {json_decode_error}')
+    raise
+except Exception as e:
+    # logs any other error
+    logging.error(f'Error, unknown error while retrieving access token. Error message: {e}')
+    raise
+
+# computes object that holds overrides file content. 
+# It uses data fetched from request
+overrides_data = {
+    "headers": {
+        "Authorization": f"Token {response_body['Token']}"
+    }
+}
+
+# log entry informing about the file override computation
+overrides_file_path = os.path.join(
+    working_directory, "api-fuzzing-overrides.json")
+logging.info("Creating overrides file: %s" % overrides_file_path)
+
+# attempts to overwrite the file
+try:
+    if os.path.exists(overrides_file_path):
+        os.unlink(overrides_file_path)
+
+    # overwrites the file with our updated dictionary
+    with open(overrides_file_path, "wb+") as fd:
+        fd.write(json.dumps(overrides_data).encode('utf-8'))
+except Exception as e:
+    # logs any other error
+    logging.error(f'Error, unkown error when overwritng file {overrides_file_path}. Error message: {e}')
+    raise
+
+# logs informing override has finished successfully
+logging.info("Override file has been updated")
+
+# end
+```
+
+In the overrides command example, the Python script depends on the `backoff` library. To make sure the library is installed before executing the Python script, the `FUZZAPI_PRE_SCRIPT` is set to a script that will install the dependencies of your overrides command.
+As for example, the following script `user-pre-scan-set-up.sh`:
+
+```shell
+#!/bin/bash
+
+# user-pre-scan-set-up.sh
+# Ensures python dependencies are installed
+
+echo "**** install python dependencies ****"
+
+python3 -m ensurepip
+pip3 install --no-cache --upgrade \
+    pip \
+    backoff
+
+echo "**** python dependencies installed ****"
+
+# end
+```
+
+You have to update your configuration to set the `FUZZAPI_PRE_SCRIPT` to our new `user-pre-scan-set-up.sh` script. For example:
+
 ```yaml
 stages:
      - fuzz
@@ -822,11 +990,14 @@ variables:
   FUZZAPI_PROFILE: Quick
   FUZZAPI_OPENAPI: test-api-specification.json
   FUZZAPI_TARGET_URL: http://test-deployment/
-  FUZZAPI_OVERRIDES_FILE: output/api-fuzzing-overrides.json
+  FUZZAPI_PRE_SCRIPT: user-pre-scan-set-up.sh
+  FUZZAPI_OVERRIDES_FILE: api-fuzzing-overrides.json
   FUZZAPI_OVERRIDES_CMD: renew_token.py
   FUZZAPI_OVERRIDES_INTERVAL: 300
 ```
 
+In the previous sample, you could use the script `user-pre-scan-set-up.sh` to also install new runtimes or applications that later on you could use in your overrides command.
+
 ### Exclude Paths
 
 > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211892) in GitLab 14.0.

doc/user/application_security/dast_api/index.md

@@ -533,7 +533,7 @@ variables:
   DAST_API_PROFILE: Quick
   DAST_API_OPENAPI: test-api-specification.json
   DAST_API_TARGET_URL: http://test-deployment/
-  DAST_API_OVERRIDES_FILE: output/dast-api-overrides.json
+  DAST_API_OVERRIDES_FILE: dast-api-overrides.json
 ```
 
 To validate that authentication is working, run an DAST API test and review the job logs and
@@ -575,13 +575,12 @@ variables:
   DAST_API_PROFILE: Quick
   DAST_API_OPENAPI: test-api-specification.json
   DAST_API_TARGET_URL: http://test-deployment/
-  DAST_API_OVERRIDES_FILE: output/dast-api-overrides.json
+  DAST_API_OVERRIDES_FILE: dast-api-overrides.json
   DAST_API_OVERRIDES_CMD: renew_token.py
   DAST_API_OVERRIDES_INTERVAL: 300
 ```
 
-To validate that authentication is working, run an DAST API test and review the job logs and
-the test API's application logs.
+To validate that authentication is working, run an DAST API test and review the job logs and the test API's application logs. See the [overrides section](#overrides) for more information about override commands.
 
 ### Configuration files
 
@@ -648,6 +647,9 @@ can be added, removed, and modified by creating a custom configuration.
 |[`DAST_API_OVERRIDES_FILE`](#overrides)                | Path to a JSON file containing overrides. |
 |[`DAST_API_OVERRIDES_ENV`](#overrides)                 | JSON string containing headers to override. |
 |[`DAST_API_OVERRIDES_CMD`](#overrides)                 | Overrides command. |
+|[`DAST_API_OVERRIDES_CMD_VERBOSE`](#overrides)         | When set to any value. It shows overrides command output as part of the job output. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334578) in GitLab 14.6. |
+|`DAST_API_PRE_SCRIPT`                                  | Run user command or script before scan session starts. |
+|`DAST_API_POST_SCRIPT`                                 | Run user command or script after scan session has finished. |
 |[`DAST_API_OVERRIDES_INTERVAL`](#overrides)            | How often to run overrides command in seconds. Defaults to `0` (once). |
 |[`DAST_API_HTTP_USERNAME`](#http-basic-authentication) | Username for HTTP authentication. |
 |[`DAST_API_HTTP_PASSWORD`](#http-basic-authentication) | Password for HTTP authentication. |
@@ -829,7 +831,7 @@ variables:
   DAST_API_PROFILE: Quick
   DAST_API_OPENAPI: test-api-specification.json
   DAST_API_TARGET_URL: http://test-deployment/
-  DAST_API_OVERRIDES_FILE: output/dast-api-overrides.json
+  DAST_API_OVERRIDES_FILE: dast-api-overrides.json
 ```
 
 #### Using a CI/CD variable
@@ -873,17 +875,29 @@ variables:
 #### Using a command
 
 If the value must be generated or regenerated on expiration, you can provide a program or script for
-the DAST API scanner to execute on a specified interval. The provided script runs in an Alpine Linux
-container that has Python 3 and Bash installed. If the Python script requires additional packages,
-it must detect this and install the packages at runtime. The script creates the overrides JSON file
-as defined above.
+the DAST API scanner to execute on a specified interval. The provided command runs in an Alpine Linux
+container that has Python 3 and Bash installed. 
+
+You have to set the environment variable `DAST_API_OVERRIDES_CMD` to the program or script you would like
+to execute. The provided command creates the overrides JSON file as defined previously.
+
+You might want to install other scripting runtimes like NodeJS or Ruby, or maybe you need to install a dependency for
+your overrides command. In this case, we recommend setting the `DAST_API_PRE_SCRIPT` to the file path of a script which
+provides those prerequisites. The script provided by `DAST_API_PRE_SCRIPT` is executed once, before the analyzer starts.
+
+See the [Alpine Linux package management](https://wiki.alpinelinux.org/wiki/Alpine_Linux_package_management) 
+page for information about installing Alpine Linux packages.
 
 You must provide three CI/CD variables, each set for correct operation:
 
 - `DAST_API_OVERRIDES_FILE`: File generated by the provided command.
-- `DAST_API_OVERRIDES_CMD`: Command to generate JSON file.
+- `DAST_API_OVERRIDES_CMD`: Overrides command in charge of generating the overrides JSON file periodically.
 - `DAST_API_OVERRIDES_INTERVAL`: Interval in seconds to run command.
 
+Optionally:
+
+- `DAST_API_PRE_SCRIPT`: Script to install runtimes or dependencies before the scan starts.
+
 ```yaml
 stages:
   - dast
@@ -895,11 +909,167 @@ variables:
   DAST_API_PROFILE: Quick
   DAST_API_OPENAPI: test-api-specification.json
   DAST_API_TARGET_URL: http://test-deployment/
-  DAST_API_OVERRIDES_FILE: output/dast-api-overrides.json
+  DAST_API_OVERRIDES_FILE: dast-api-overrides.json
   DAST_API_OVERRIDES_CMD: renew_token.py
   DAST_API_OVERRIDES_INTERVAL: 300
 ```
 
+#### Debugging overrides
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334578) in GitLab 14.8.
+
+By default the output of the overrides command is hidden. If the overrides command returns a non zero exit code, the command is displayed as part of your job output. Optionally, you can set the variable `DAST_API_OVERRIDES_CMD_VERBOSE` to any value in order to display overrides command output as it is generated. This is useful when testing your overrides script, but should be disabled afterwards as it slows down testing.
+
+It is also possible to write messages from your script to a log file that is collected when the job completes or fails. The log file must be created in a specific location and following a naming convention.
+
+Adding some basic logging to your overrides script is useful in case the script fails unexpectedly during normal running of the job. The log file is automatically included as an artifact of the job, allowing you to download it after the job has finished.
+
+Following our example, we provided `renew_token.py` in the environment variable `DAST_API_OVERRIDES_CMD`. Please notice two things in the script:
+
+- Log file is saved in the location indicated by the environmental variable `CI_PROJECT_DIR`.
+- Log file name should match `gl-*.log`.
+
+```python
+#!/usr/bin/env python
+
+# Example of an overrides command
+
+# Override commands can update the overrides json file
+# with new values to be used.  This is a great way to
+# update an authentication token that will expire
+# during testing.
+
+import logging
+import json
+import os
+import requests
+import backoff
+
+# [1] Store log file in directory indicated by env var CI_PROJECT_DIR
+working_directory = os.environ['CI_PROJECT_DIR']
+
+# [2] File name should match the pattern: gl-*.log
+log_file_path = os.path.join(working_directory, 'gl-user-overrides.log')
+
+# Set up logger
+logging.basicConfig(filename=log_file_path, level=logging.DEBUG)
+
+# Use `backoff` decorator to retry in case of transient errors.
+@backoff.on_exception(backoff.expo,
+                      (requests.exceptions.Timeout,
+                       requests.exceptions.ConnectionError),
+                       max_time=30)
+def get_auth_response():
+    return requests.get('https://authorization.service/api/get_api_token', auth=(os.environ['AUTH_USER'], os.environ['AUTH_PWD']))
+
+# In our example, access token is retrieved from a given endpoint
+try:
+
+    # Performs a http request, response sample: 
+    # { "Token" : "b5638ae7-6e77-4585-b035-7d9de2e3f6b3" }
+    response = get_auth_response()
+    
+    # Check that the request is successful. may raise `requests.exceptions.HTTPError`
+    response.raise_for_status()
+
+    # Gets JSON data
+    response_body = response.json()
+
+# If needed specific exceptions can be caught
+# requests.ConnectionError                  : A network connection error problem occurred
+# requests.HTTPError                        : HTTP request returned an unsuccessful status code. [Response.raise_for_status()]
+# requests.ConnectTimeout                   : The request timed out while trying to connect to the remote server
+# requests.ReadTimeout                      : The server did not send any data in the allotted amount of time.
+# requests.TooManyRedirects                 : The request exceeds the configured number of maximum redirections
+# requests.exceptions.RequestException      : All exceptions that related to Requests
+except requests.exceptions.RequestException as requests_error:
+    # logs  exceptions  related to `Requests`
+    logging.error(f'Error, failed while performing HTTP request. Error message: {requests_error}')
+    raise
+except requests.exceptions.JSONDecodeError as json_decode_error:
+    # logs errors related decoding JSON response
+    logging.error(f'Error, failed while decoding JSON response. Error message: {json_decode_error}')
+    raise
+except Exception as e:
+    # logs any other error
+    logging.error(f'Error, unknown error while retrieving access token. Error message: {e}')
+    raise
+
+# computes object that holds overrides file content. 
+# It uses data fetched from request
+overrides_data = {
+    "headers": {
+        "Authorization": f"Token {response_body['Token']}"
+    }
+}
+
+# log entry informing about the file override computation
+# the location of the overrides json file is also CI_PROJECT_DIR
+overrides_file_path = os.path.join(
+    working_directory, "dast-api-overrides.json")
+logging.info("Creating overrides file: %s" % overrides_file_path)
+
+# attempts to overwrite the file
+try:
+    if os.path.exists(overrides_file_path):
+        os.unlink(overrides_file_path)
+
+    # overwrites the file with our updated dictionary
+    with open(overrides_file_path, "wb+") as fd:
+        fd.write(json.dumps(overrides_data).encode('utf-8'))
+except Exception as e:
+    # logs any other error
+    logging.error(f'Error, unkown error when overwritng file {overrides_file_path}. Error message: {e}')
+    raise
+
+# logs informing override has finished successfully
+logging.info("Override file has been updated")
+
+# end
+```
+
+In the overrides command example, the Python script depends on the `backoff` library. To make sure the library is installed before executing the Python script, the `DAST_API_PRE_SCRIPT` is set to a script that will install the dependencies of your overrides command.
+As for example, the following script `user-pre-scan-set-up.sh`
+
+```shell
+#!/bin/bash
+
+# user-pre-scan-set-up.sh
+# Ensures python dependencies are installed
+
+echo "**** install python dependencies ****"
+
+python3 -m ensurepip
+pip3 install --no-cache --upgrade \
+    pip \
+    backoff
+
+echo "**** python dependencies installed ****"
+
+# end
+```
+
+You have to update your configuration to set the `DAST_API_PRE_SCRIPT` to our new `user-pre-scan-set-up.sh` script. For example:
+
+```yaml
+stages:
+  - dast
+
+include:
+  - template: DAST-API.gitlab-ci.yml
+
+variables:
+  DAST_API_PROFILE: Quick
+  DAST_API_OPENAPI: test-api-specification.json
+  DAST_API_TARGET_URL: http://test-deployment/
+  DAST_API_PRE_SCRIPT: user-pre-scan-set-up.sh
+  DAST_API_OVERRIDES_FILE: dast-api-overrides.json
+  DAST_API_OVERRIDES_CMD: renew_token.py
+  DAST_API_OVERRIDES_INTERVAL: 300
+```
+
+In the previous sample, you could use the script `user-pre-scan-set-up.sh` to also install new runtimes or applications that later on you could use in our overrides command.
+
 ### Exclude Paths
 
 > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211892) in GitLab 14.0.