Merge branch 'docs/ci-caching' into 'master'

Add docs on CI caching

Closes #32515, #18945, and gitlab-runner#1621

See merge request gitlab-org/gitlab-ce!12183

doc/ci/README.md

@@ -65,7 +65,8 @@ learn how to leverage its potential even more.
   environments and use them for different purposes like testing, building and
   deploying
 - [Job artifacts](../user/project/pipelines/job_artifacts.md)
-- [Git submodules](git_submodules.md): How to run your CI jobs when Git
+- [Caching dependencies](caching/index.md)
+- [Git submodules](git_submodules.md) - How to run your CI jobs when Git
   submodules are involved
 - [Use SSH keys in your build environment](ssh_keys/README.md)
 - [Trigger pipelines through the GitLab API](triggers/README.md)

doc/ci/runners/README.md

@@ -146,24 +146,7 @@ To protect/unprotect Runners:
 
 ## Manually clearing the Runners cache
 
-> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/41249) in GitLab 10.4.
-
-GitLab Runners use [cache](../yaml/README.md#cache) to speed up the execution
-of your jobs by reusing existing data. This however, can sometimes lead to an
-inconsistent behavior.
-
-To start with a fresh copy of the cache, you can easily do it via GitLab's UI:
-
-1. Navigate to your project's **CI/CD > Pipelines** page.
-1. Click on the **Clear Runner caches** to clean up the cache.
-1. On the next push, your CI/CD job will use a new cache.
-
-That way, you don't have to change the [cache key](../yaml/README.md#cache-key)
-in your `.gitlab-ci.yml`.
-
-Behind the scenes, this works by increasing a counter in the database, and the
-value of that counter is used to create the key for the cache. After a push, a
-new key is generated and the old cache is not valid anymore.
+Read [clearing the cache](../caching/index.md#clearing-the-cache).
 
 ## How shared Runners pick jobs
 
@@ -227,15 +210,16 @@ that it may encounter on the projects it's shared over. This would be
 problematic for large amounts of projects, if it wasn't for tags.
 
 By tagging a Runner for the types of jobs it can handle, you can make sure
-shared Runners will only run the jobs they are equipped to run.
+shared Runners will [only run the jobs they are equipped to run](../yaml/README.md#tags).
 
 For instance, at GitLab we have Runners tagged with "rails" if they contain
 the appropriate dependencies to run Rails test suites.
 
 ### Preventing Runners with tags from picking jobs without tags
 
-You can configure a Runner to prevent it from picking jobs with tags when
-the Runner does not have tags assigned. This setting can be enabled the first
+You can configure a Runner to prevent it from picking
+[jobs with tags](../yaml/README.md#tags) when the Runner does not have tags
+assigned. This setting can be enabled the first
 time you [register a Runner][register] and can be changed afterwards under
 each Runner's settings.
 

doc/ci/yaml/README.md

@@ -674,6 +674,10 @@ as Review Apps. You can see a simple example using Review Apps at
   by default.
 - From GitLab 9.2, caches are restored before [artifacts](#artifacts).
 
+TIP: **Learn more:**
+Read how caching works and find out some good practices in the
+[caching dependencies documentation](../caching/index.md).
+
 `cache` is used to specify a list of files and directories which should be
 cached between jobs. You can only use paths that are within the project
 workspace.
@@ -681,35 +685,20 @@ workspace.
 If `cache` is defined outside the scope of jobs, it means it is set
 globally and all jobs will use that definition.
 
-Cache all files in `binaries` and `.config`:
-
-```yaml
-rspec:
-  script: test
-  cache:
-    paths:
-    - binaries/
-    - .config
-```
-
-Cache all Git untracked files:
+### `cache:paths`
 
-```yaml
-rspec:
-  script: test
-  cache:
-    untracked: true
-```
+Use the `paths` directive to choose which files or directories will be cached.
+Wildcards can be used as well.
 
-Cache all Git untracked files and files in `binaries`:
+Cache all files in `binaries` that end in `.apk` and the `.config` file:
 
 ```yaml
 rspec:
   script: test
   cache:
-    untracked: true
     paths:
-    - binaries/
+    - binaries/*.apk
+    - .config
 ```
 
 Locally defined cache overrides globally defined options. The following `rspec`
@@ -723,33 +712,26 @@ cache:
 rspec:
   script: test
   cache:
-    key: rspec
     paths:
     - binaries/
 ```
 
-Note that since cache is shared between jobs, if you're using different
-paths for different jobs, you should also set a different **cache:key**
-otherwise cache content can be overwritten.
-
-NOTE: **Note:**
-The cache is provided on a best-effort basis, so don't expect that the cache
-will be always present.
-
 ### `cache:key`
 
 > Introduced in GitLab Runner v1.0.0.
 
-The `key` directive allows you to define the affinity of caching
-between jobs, allowing to have a single cache for all jobs,
-cache per-job, cache per-branch or any other way that fits your needs.
+Since the cache is shared between jobs, if you're using different
+paths for different jobs, you should also set a different `cache:key`
+otherwise cache content can be overwritten.
 
-This way, you can fine tune caching, allowing you to cache data between
-different jobs or even different branches.
+The `key` directive allows you to define the affinity of caching between jobs,
+allowing to have a single cache for all jobs, cache per-job, cache per-branch
+or any other way that fits your workflow. This way, you can fine tune caching,
+allowing you to cache data between different jobs or even different branches.
 
 The `cache:key` variable can use any of the
 [predefined variables](../variables/README.md), and the default key, if not set,
-is set as `$CI_JOB_NAME-$CI_COMMIT_REF_NAME` which translates as "per-job and
+is `$CI_JOB_NAME-$CI_COMMIT_REF_NAME` which translates as "per-job and
 per-branch". It is the default across the project, therefore everything is
 shared between pipelines and jobs running on the same branch by default.
 
@@ -757,56 +739,56 @@ NOTE: **Note:**
 The `cache:key` variable cannot contain the `/` character, or the equivalent
 URI-encoded `%2F`; a value made only of dots (`.`, `%2E`) is also forbidden.
 
-**Example configurations**
-
-To enable per-job caching:
-
-```yaml
-cache:
-  key: "$CI_JOB_NAME"
-  untracked: true
-```
-
-To enable per-branch caching:
+For example, to enable per-branch caching:
 
 ```yaml
 cache:
   key: "$CI_COMMIT_REF_SLUG"
-  untracked: true
+  paths:
+  - binaries/
 ```
 
-To enable per-job and per-branch caching:
+If you use **Windows Batch** to run your shell scripts you need to replace
+`$` with `%`:
 
 ```yaml
 cache:
-  key: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
-  untracked: true
+  key: "%CI_JOB_STAGE%-%CI_COMMIT_REF_SLUG%"
+  paths:
+  - binaries/
 ```
 
-To enable per-branch and per-stage caching:
+If you use **Windows PowerShell** to run your shell scripts you need to replace
+`$` with `$env:`:
 
 ```yaml
 cache:
-  key: "$CI_JOB_STAGE-$CI_COMMIT_REF_SLUG"
-  untracked: true
+  key: "$env:CI_JOB_STAGE-$env:CI_COMMIT_REF_SLUG"
+  paths:
+  - binaries/
 ```
 
-If you use **Windows Batch** to run your shell scripts you need to replace
-`$` with `%`:
+### `cache:untracked`
+
+Set `untracked: true` to cache all files that are untracked in your Git
+repository:
 
 ```yaml
-cache:
-  key: "%CI_JOB_STAGE%-%CI_COMMIT_REF_SLUG%"
-  untracked: true
+rspec:
+  script: test
+  cache:
+    untracked: true
 ```
 
-If you use **Windows PowerShell** to run your shell scripts you need to replace
-`$` with `$env:`:
+Cache all Git untracked files and files in `binaries`:
 
 ```yaml
-cache:
-  key: "$env:CI_JOB_STAGE-$env:CI_COMMIT_REF_SLUG"
-  untracked: true
+rspec:
+  script: test
+  cache:
+    untracked: true
+    paths:
+    - binaries/
 ```
 
 ### `cache:policy`