Merge branch 'caches' into 'master'

Allow to define cache in `.gitlab-ci.yml`

This extends `.gitlab-ci.yml` syntax to allow specifying caching files and directories between builds, making it easy to preserve ex. gems.

```
cache:
  paths:
  - .bundle
  - vendor/

before_script:
- bundle install --path vendor/

rspec:
  script:
  - bundle exec rspec
```

This is based on Build Artifacts changes.

/cc@dzaporozhets 

See merge request !1786

CHANGELOG

@@ -20,6 +20,7 @@ v 8.2.0 (unreleased)
   - Send build name and stage in CI notification e-mail
   - Extend yml syntax for only and except to support specifying repository path
   - Enable shared runners to all new projects
+  - Allow to define cache in `.gitlab-ci.yml`
   - Fix: 500 error returned if destroy request without HTTP referer (Kazuki Shimizu)
   - Remove deprecated CI events from project settings page
   - [API] Add ability to fetch the commit ID of the last commit that actually touched a file

doc/ci/yaml/README.md

@@ -56,6 +56,7 @@ There are a few `keywords` that can't be used as job names:
 | types         | optional | Alias for `stages` |
 | before_script | optional | Define commands prepended for each job's script |
 | variables     | optional | Define build variables |
+| cache         | optional | Define list of files that should be cached between subsequent runs |
 
 ### image and services
 This allows to specify a custom Docker image and a list of services that can be used for time of the build.
@@ -110,6 +111,19 @@ These variables can be later used in all executed commands and scripts.
 
 The YAML-defined variables are also set to all created service containers, thus allowing to fine tune them.
 
+### cache
+`cache` is used to specify list of files and directories which should be cached between builds.
+
+**The global setting allows to specify default cached files for all jobs.**
+
+To cache all git untracked files and files in `binaries`:
+```
+cache:
+  untracked: true
+  paths:
+  - binaries/
+```
+
 ## Jobs
 `.gitlab-ci.yml` allows you to specify an unlimited number of jobs.
 Each job has to have a unique `job_name`, which is not one of the keywords mentioned above.
@@ -142,6 +156,7 @@ job_name:
 | allow_failure | optional | Allow build to fail. Failed build doesn't contribute to commit status |
 | when          | optional | Define when to run build. Can be `on_success`, `on_failure` or `always` |
 | artifacts     | optional | Define list build artifacts |
+| cache         | optional | Define list of files that should be cached between subsequent runs |
 
 ### script
 `script` is a shell script which is executed by runner. The shell script is prepended with `before_script`.
@@ -288,6 +303,57 @@ The artifacts will be send after the build success to GitLab and will be accessi
 
 This feature requires GitLab Runner v0.7.0 or higher.
 
+### cache
+`cache` is used to specify list of files and directories which should be cached between builds.
+
+1. Cache all files in `binaries` and `.config`:
+```
+rspec:
+  script: test
+  cache:
+    paths:
+    - binaries/
+    - .config
+```
+
+2. Cache all git untracked files:
+```
+rspec:
+  script: test
+  cache:
+    untracked: true
+```
+
+3. Cache all git untracked files and files in `binaries`:
+```
+rspec:
+  script: test
+  cache:
+    untracked: true
+    paths:
+    - binaries/
+```
+
+4. Locally defined cache overwrites globally defined options. This will cache only `binaries/`:
+
+```
+cache:
+  paths:
+  - my/files
+
+rspec:
+  script: test
+  cache:
+    paths:
+    - binaries/
+```
+
+The cache is provided on best effort basis, so don't expect that cache will be present.
+For implementation details please check GitLab Runner.
+
+This feature requires GitLab Runner v0.7.0 or higher.
+
+
 ## Validate the .gitlab-ci.yml
 Each instance of GitLab CI has an embedded debug tool called Lint.
 You can find the link to the Lint in the project's settings page or use short url `/lint`.

lib/ci/gitlab_ci_yaml_processor.rb

@@ -4,10 +4,10 @@ module Ci
 
     DEFAULT_STAGES = %w(build test deploy)
     DEFAULT_STAGE = 'test'
-    ALLOWED_YAML_KEYS = [:before_script, :image, :services, :types, :stages, :variables]
-    ALLOWED_JOB_KEYS = [:tags, :script, :only, :except, :type, :image, :services, :allow_failure, :type, :stage, :when, :artifacts]
+    ALLOWED_YAML_KEYS = [:before_script, :image, :services, :types, :stages, :variables, :cache]
+    ALLOWED_JOB_KEYS = [:tags, :script, :only, :except, :type, :image, :services, :allow_failure, :type, :stage, :when, :artifacts, :cache]
 
-    attr_reader :before_script, :image, :services, :variables, :path
+    attr_reader :before_script, :image, :services, :variables, :path, :cache
 
     def initialize(config, path = nil)
       @config = YAML.load(config)
@@ -46,6 +46,7 @@ module Ci
       @services = @config[:services]
       @stages = @config[:stages] || @config[:types]
       @variables = @config[:variables] || {}
+      @cache = @config[:cache]
       @config.except!(*ALLOWED_YAML_KEYS)
 
       # anything that doesn't have script is considered as unknown
@@ -78,7 +79,8 @@ module Ci
         options: {
           image: job[:image] || @image,
           services: job[:services] || @services,
-          artifacts: job[:artifacts]
+          artifacts: job[:artifacts],
+          cache: job[:cache] || @cache,
         }.compact
       }
     end
@@ -112,6 +114,16 @@ module Ci
         raise ValidationError, "variables should be a map of key-valued strings"
       end
 
+      if @cache
+        if @cache[:untracked] && !validate_boolean(@cache[:untracked])
+          raise ValidationError, "cache:untracked parameter should be an boolean"
+        end
+
+        if @cache[:paths] && !validate_array_of_strings(@cache[:paths])
+          raise ValidationError, "cache:paths parameter should be an array of strings"
+        end
+      end
+
       @jobs.each do |name, job|
         validate_job!(name, job)
       end
@@ -160,6 +172,16 @@ module Ci
         raise ValidationError, "#{name} job: except parameter should be an array of strings"
       end
 
+      if job[:cache]
+        if job[:cache][:untracked] && !validate_boolean(job[:cache][:untracked])
+          raise ValidationError, "#{name} job: cache:untracked parameter should be an boolean"
+        end
+
+        if job[:cache][:paths] && !validate_array_of_strings(job[:cache][:paths])
+          raise ValidationError, "#{name} job: cache:paths parameter should be an array of strings"
+        end
+      end
+
       if job[:artifacts]
         if job[:artifacts][:untracked] && !validate_boolean(job[:artifacts][:untracked])
           raise ValidationError, "#{name} job: artifacts:untracked parameter should be an boolean"

spec/lib/ci/gitlab_ci_yaml_processor_spec.rb

@@ -333,6 +333,60 @@ module Ci
       end
     end
 
+    describe "Caches" do
+      it "returns cache when defined globally" do
+        config = YAML.dump({
+                             cache: { paths: ["logs/", "binaries/"], untracked: true },
+                             rspec: {
+                               script: "rspec"
+                             }
+                           })
+
+        config_processor = GitlabCiYamlProcessor.new(config)
+
+        expect(config_processor.builds_for_stage_and_ref("test", "master").size).to eq(1)
+        expect(config_processor.builds_for_stage_and_ref("test", "master").first[:options][:cache]).to eq(
+          paths: ["logs/", "binaries/"],
+          untracked: true,
+        )
+      end
+
+      it "returns cache when defined in a job" do
+        config = YAML.dump({
+                             rspec: {
+                               cache: { paths: ["logs/", "binaries/"], untracked: true },
+                               script: "rspec"
+                             }
+                           })
+
+        config_processor = GitlabCiYamlProcessor.new(config)
+
+        expect(config_processor.builds_for_stage_and_ref("test", "master").size).to eq(1)
+        expect(config_processor.builds_for_stage_and_ref("test", "master").first[:options][:cache]).to eq(
+          paths: ["logs/", "binaries/"],
+          untracked: true,
+        )
+      end
+
+      it "overwrite cache when defined for a job and globally" do
+        config = YAML.dump({
+                             cache: { paths: ["logs/", "binaries/"], untracked: true },
+                             rspec: {
+                               script: "rspec",
+                               cache: { paths: ["test/"], untracked: false },
+                             }
+                           })
+
+        config_processor = GitlabCiYamlProcessor.new(config)
+
+        expect(config_processor.builds_for_stage_and_ref("test", "master").size).to eq(1)
+        expect(config_processor.builds_for_stage_and_ref("test", "master").first[:options][:cache]).to eq(
+          paths: ["test/"],
+          untracked: false,
+        )
+      end
+    end
+
     describe "Artifacts" do
       it "returns artifacts when defined" do
         config = YAML.dump({
@@ -542,6 +596,34 @@ module Ci
           GitlabCiYamlProcessor.new(config)
         end.to raise_error(GitlabCiYamlProcessor::ValidationError, "rspec job: artifacts:paths parameter should be an array of strings")
       end
+
+      it "returns errors if cache:untracked is not an array of strings" do
+        config = YAML.dump({ cache: { untracked: "string" }, rspec: { script: "test" } })
+        expect do
+          GitlabCiYamlProcessor.new(config)
+        end.to raise_error(GitlabCiYamlProcessor::ValidationError, "cache:untracked parameter should be an boolean")
+      end
+
+      it "returns errors if cache:paths is not an array of strings" do
+        config = YAML.dump({ cache: { paths: "string" }, rspec: { script: "test" } })
+        expect do
+          GitlabCiYamlProcessor.new(config)
+        end.to raise_error(GitlabCiYamlProcessor::ValidationError, "cache:paths parameter should be an array of strings")
+      end
+
+      it "returns errors if job cache:untracked is not an array of strings" do
+        config = YAML.dump({ types: ["build", "test"], rspec: { script: "test", cache: { untracked: "string" } } })
+        expect do
+          GitlabCiYamlProcessor.new(config)
+        end.to raise_error(GitlabCiYamlProcessor::ValidationError, "rspec job: cache:untracked parameter should be an boolean")
+      end
+
+      it "returns errors if job cache:paths is not an array of strings" do
+        config = YAML.dump({ types: ["build", "test"], rspec: { script: "test", cache: { paths: "string" } } })
+        expect do
+          GitlabCiYamlProcessor.new(config)
+        end.to raise_error(GitlabCiYamlProcessor::ValidationError, "rspec job: cache:paths parameter should be an array of strings")
+      end
     end
   end
 end