Merge branch 'ce-to-ee-2018-06-27' into 'master'

CE upstream - 2018-06-27 18:21 UTC

Closes gitaly#1237 and gitaly#1218

See merge request gitlab-org/gitlab-ee!6313

GITALY_SERVER_VERSION

-0.108.0
+0.109.0

PROCESS.md

@@ -205,24 +205,7 @@ to. For example:
 If you think a merge request should go into an RC or patch even though it does not meet these requirements,
 you can ask for an exception to be made.
 
-Go to [Release tasks issue tracker](https://gitlab.com/gitlab-org/release/tasks/issues/new) and create an issue
-using the `Exception-request` issue template.
-
-**Do not** set the relevant `Pick into X.Y` label (see above) before request an
-exception; this should be done after the exception is approved.
-
-You can find who is who on the [team page](https://about.gitlab.com/team/).
-
-Whether an exception is made is determined by weighing the benefit and urgency of the change
-(how important it is to the company that this is released _right now_ instead of in a month)
-against the potential negative impact
-(things breaking without enough time to comfortably find and fix them before the release on the 22nd).
-When in doubt, we err on the side of _not_ cherry-picking.
-
-For example, it is likely that an exception will be made for a trivial 1-5 line performance improvement
-(e.g. adding a database index or adding `includes` to a query), but not for a new feature, no matter how relatively small or thoroughly tested.
-
-All MRs which have had exceptions granted must be merged by the 15th.
+Check [this guide](https://gitlab.com/gitlab-org/release/docs/blob/master/general/exception-request/process.md) about how to open an exception request before opening one.
 
 ### Regressions
 

app/helpers/application_helper.rb

@@ -276,7 +276,7 @@ module ApplicationHelper
     {
       members: members_project_autocomplete_sources_path(object, type: noteable_type, type_id: params[:id]),
       issues: issues_project_autocomplete_sources_path(object),
-      merge_requests: merge_requests_project_autocomplete_sources_path(object),
+      mergeRequests: merge_requests_project_autocomplete_sources_path(object),
       labels: labels_project_autocomplete_sources_path(object, type: noteable_type, type_id: params[:id]),
       milestones: milestones_project_autocomplete_sources_path(object),
       commands: commands_project_autocomplete_sources_path(object, type: noteable_type, type_id: params[:id])

app/workers/ci/archive_traces_cron_worker.rb

+# frozen_string_literal: true
+
 module Ci
   class ArchiveTracesCronWorker
     include ApplicationWorker

app/workers/ci/build_trace_chunk_flush_worker.rb

+# frozen_string_literal: true
+
 module Ci
   class BuildTraceChunkFlushWorker
     include ApplicationWorker

app/workers/concerns/application_worker.rb

+# frozen_string_literal: true
+
 Sidekiq::Worker.extend ActiveSupport::Concern
 
 module ApplicationWorker

app/workers/concerns/cluster_applications.rb

+# frozen_string_literal: true
+
 module ClusterApplications
   extend ActiveSupport::Concern
 

app/workers/concerns/cluster_queue.rb

+# frozen_string_literal: true
+
 ##
 # Concern for setting Sidekiq settings for the various Gcp clusters workers.
 #

app/workers/concerns/cronjob_queue.rb

+# frozen_string_literal: true
+
 # Concern that sets various Sidekiq settings for workers executed using a
 # cronjob.
 module CronjobQueue

app/workers/concerns/exception_backtrace.rb

+# frozen_string_literal: true
+
 # Concern for enabling a few lines of exception backtraces in Sidekiq
 module ExceptionBacktrace
   extend ActiveSupport::Concern

app/workers/concerns/gitlab/github_import/queue.rb

+# frozen_string_literal: true
+
 module Gitlab
   module GithubImport
     module Queue

app/workers/concerns/mail_scheduler_queue.rb

+# frozen_string_literal: true
+
 module MailSchedulerQueue
   extend ActiveSupport::Concern
 

app/workers/concerns/new_issuable.rb

+# frozen_string_literal: true
+
 module NewIssuable
   attr_reader :issuable, :user
 

app/workers/concerns/object_storage_queue.rb

+# frozen_string_literal: true
+
 # Concern for setting Sidekiq settings for the various GitLab ObjectStorage workers.
 module ObjectStorageQueue
   extend ActiveSupport::Concern

app/workers/concerns/pipeline_background_queue.rb

+# frozen_string_literal: true
+
 ##
 # Concern for setting Sidekiq settings for the low priority CI pipeline workers.
 #

app/workers/concerns/pipeline_queue.rb

+# frozen_string_literal: true
+
 ##
 # Concern for setting Sidekiq settings for the various CI pipeline workers.
 #

app/workers/concerns/project_import_options.rb

+# frozen_string_literal: true
+
 module ProjectImportOptions
   extend ActiveSupport::Concern
 

app/workers/concerns/project_start_import.rb

+# frozen_string_literal: true
+
 # Used in EE by mirroring
 module ProjectStartImport
   def start(project)

app/workers/concerns/repository_check_queue.rb

+# frozen_string_literal: true
+
 # Concern for setting Sidekiq settings for the various repository check workers.
 module RepositoryCheckQueue
   extend ActiveSupport::Concern

app/workers/concerns/waitable_worker.rb

+# frozen_string_literal: true
+
 module WaitableWorker
   extend ActiveSupport::Concern
 

app/workers/mail_scheduler/issue_due_worker.rb

+# frozen_string_literal: true
+
 module MailScheduler
   class IssueDueWorker
     include ApplicationWorker

app/workers/mail_scheduler/notification_service_worker.rb

+# frozen_string_literal: true
+
 require 'active_job/arguments'
 
 module MailScheduler

app/workers/object_storage/background_move_worker.rb

+# frozen_string_literal: true
+
 module ObjectStorage
   class BackgroundMoveWorker
     include ApplicationWorker

app/workers/repository_check/batch_worker.rb

+# frozen_string_literal: true
+
 module RepositoryCheck
   class BatchWorker
     prepend ::EE::RepositoryCheck::BatchWorker

app/workers/repository_check/clear_worker.rb

+# frozen_string_literal: true
+
 module RepositoryCheck
   class ClearWorker
     include ApplicationWorker

app/workers/repository_check/single_repository_worker.rb

+# frozen_string_literal: true
+
 module RepositoryCheck
   class SingleRepositoryWorker
     include ApplicationWorker

changelogs/unreleased/46546-do-not-pre-select-previous-user-s-when-creating-protected-branches.yml

+---
+title: CE port gitlab-ee!6112
+merge_request: 19714
+author:
+type: other

changelogs/unreleased/48528-fix-mr-autocompletion.yml

+---
+title: Fix broken '!' support to autocomplete MRs in GFM fields
+merge_request: 20204
+author:
+type: fixed

changelogs/unreleased/frozen-string-enable-app-workers-2.yml

+---
+title: Finish enabling frozen string for app/workers/*.rb
+merge_request: 20197
+author: gfyoung
+type: other

config/initializers/1_settings.rb

@@ -454,7 +454,7 @@ repositories_storages          = Settings.repositories.storages.values
 repository_downloads_path      = Settings.gitlab['repository_downloads_path'].to_s.gsub(%r{/$}, '')
 repository_downloads_full_path = File.expand_path(repository_downloads_path, Settings.gitlab['user_home'])
 
-# Gitaly migration: https://gitlab.com/gitlab-org/gitaly/issues/1237
+# Gitaly migration: https://gitlab.com/gitlab-org/gitaly/issues/1255
 Gitlab::GitalyClient::StorageSettings.allow_disk_access do
   if repository_downloads_path.blank? || repositories_storages.any? { |rs| [repository_downloads_path, repository_downloads_full_path].include?(rs.legacy_disk_path.gsub(%r{/$}, '')) }
     Settings.gitlab['repository_downloads_path'] = File.join(Settings.shared['path'], 'cache/archive')

config/initializers/6_validations.rb

@@ -2,20 +2,6 @@ def storage_name_valid?(name)
   !!(name =~ /\A[a-zA-Z0-9\-_]+\z/)
 end
 
-def find_parent_path(name, path)
-  parent = Pathname.new(path).realpath.parent
-  Gitlab.config.repositories.storages.detect do |n, rs|
-    name != n && Pathname.new(rs.legacy_disk_path).realpath == parent
-  end
-rescue Errno::EIO, Errno::ENOENT => e
-  warning = "WARNING: couldn't verify #{path} (#{name}). "\
-            "If this is an external storage, it might be offline."
-  message = "#{warning}\n#{e.message}"
-  Rails.logger.error("#{message}\n\t" + e.backtrace.join("\n\t"))
-
-  nil
-end
-
 def storage_validation_error(message)
   raise "#{message}. Please fix this in your gitlab.yml before starting GitLab."
 end
@@ -37,17 +23,4 @@ def validate_storages_config
   end
 end
 
-# Gitaly migration: https://gitlab.com/gitlab-org/gitaly/issues/1237
-def validate_storages_paths
-  Gitlab::GitalyClient::StorageSettings.allow_disk_access do
-    Gitlab.config.repositories.storages.each do |name, repository_storage|
-      parent_name, _parent_path = find_parent_path(name, repository_storage.legacy_disk_path)
-      if parent_name
-        storage_validation_error("#{name} is a nested path of #{parent_name}. Nested paths are not supported for repository storages")
-      end
-    end
-  end
-end
-
 validate_storages_config
-validate_storages_paths unless Rails.env.test? || ENV['SKIP_STORAGE_VALIDATION'] == 'true'

doc/administration/repository_storage_types.md

@@ -82,6 +82,46 @@ To migrate your existing projects to the new storage type, check the specific
 [rake tasks]: raketasks/storage.md#migrate-existing-projects-to-hashed-storage
 [storage-paths]: repository_storage_types.md
 
+#### Rollback
+
+There is no automated rollback implemented. Below are the steps required to rollback
+from each storage migration.
+
+The rollback has to be performed in the reverse order. To get into "Legacy" state,
+you need to rollback Attachments first, then Project.
+
+Also note that if Geo is enabled, after the migration was triggered, an event is generated
+to replicate the operation on any Secondary node. That means the on disk changes will also
+need to be performed on these nodes as well. Database changes will propagate without issues.
+
+You must make sure the migration event was already processed or otherwise it may migrate
+the files back to Hashed state again.
+
+##### Attachments
+
+To rollback single Attachment migration, rename `aa/bb/abcdef1234567890...` folder back to `namespace/project`.
+
+Both folder names can be generated by the `FileUploader.absolute_base_dir(project)`, you
+just need to switch the version from the `project` back to the previous one.
+
+```ruby
+project.storage_version
+# => 2
+
+FileUploader.absolute_base_dir(project)
+# => "/opt/gitlab/embedded/service/gitlab-rails/public/uploads/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35"
+
+project.storage_version = 1
+
+FileUploader.absolute_base_dir(project)
+# => "/opt/gitlab/embedded/service/gitlab-rails/public/uploads/gitlab/gitlab-shell-renamed"
+```
+
+##### Project
+
+To rollback single Project migration, move `@hashed/aa/bb/aabbcdef1234567890abcdef.git` and `@hashed/aa/bb/aabbcdef1234567890abcdef.wiki.git`
+back to `namespace/project.git` and `namespace/project.wiki.git` respectively and switch the version from the `project` back to `null`.
+
 ### Hashed Storage coverage
 
 We are incrementally moving every storable object in GitLab to the Hashed
@@ -100,6 +140,30 @@ which is true for CI Cache and LFS Objects.
 | Pages           | Yes            | No             | -             | -              |
 | Docker Registry | Yes            | No             | -             | -              |
 | CI Build Logs   | No             | No             | -             | -              |
-| CI Artifacts    | No             | No             | Yes (Premium) | -              |
+| CI Artifacts    | No             | No             | Yes           | 9.4 / 10.6     |
 | CI Cache        | No             | No             | Yes           | -              |
-| LFS Objects     | Yes            | No             | Yes (Premium) | -              |
+| LFS Objects     | Yes            | Similar        | Yes           | 10.0 / 10.7    |
+
+#### Implementation Details
+
+##### Avatars
+
+Each file is stored in a folder with its `id` from the database. The filename is always `avatar.png` for user avatars.
+When avatar is replaced, `Upload` model is destroyed and a new one takes place with different `id`.
+
+##### CI Artifacts
+
+CI Artifacts are S3 compatible since **9.4** (GitLab Premium), and available in GitLab Core since **10.6**.
+
+##### LFS Objects
+
+LFS Objects implements a similar storage pattern using 2 chars, 2 level folders, following git own implementation:
+
+```ruby
+"shared/lfs-objects/#{oid[0..1}/#{oid[2..3]}/#{oid[4..-1]}"
+
+# Based on object `oid`: `8909029eb962194cfb326259411b22ae3f4a814b5be4f80651735aeef9f3229c`, path will be:
+"shared/lfs-objects/89/09/029eb962194cfb326259411b22ae3f4a814b5be4f80651735aeef9f3229c"
+```
+
+They are also S3 compatible since **10.0** (GitLab Premium), and available in GitLab Core since **10.7**.

spec/helpers/application_helper_spec.rb

@@ -157,7 +157,7 @@ describe ApplicationHelper do
     let(:noteable_type) { Issue }
     it 'returns paths for autocomplete_sources_controller' do
       sources = helper.autocomplete_data_sources(project, noteable_type)
-      expect(sources.keys).to match_array([:members, :issues, :merge_requests, :labels, :milestones, :commands])
+      expect(sources.keys).to match_array([:members, :issues, :mergeRequests, :labels, :milestones, :commands])
       sources.keys.each do |key|
         expect(sources[key]).not_to be_nil
       end

spec/initializers/6_validations_spec.rb

@@ -44,49 +44,6 @@ describe '6_validations' do
     end
   end
 
-  describe 'validate_storages_paths' do
-    context 'with correct settings' do
-      before do
-        mock_storages('foo' => Gitlab::GitalyClient::StorageSettings.new('path' => 'tmp/tests/paths/a/b/c'), 'bar' => Gitlab::GitalyClient::StorageSettings.new('path' => 'tmp/tests/paths/a/b/d'))
-      end
-
-      it 'passes through' do
-        expect { validate_storages_paths }.not_to raise_error
-      end
-    end
-
-    context 'with nested storage paths' do
-      before do
-        mock_storages('foo' => Gitlab::GitalyClient::StorageSettings.new('path' => 'tmp/tests/paths/a/b/c'), 'bar' => Gitlab::GitalyClient::StorageSettings.new('path' => 'tmp/tests/paths/a/b/c/d'))
-      end
-
-      it 'throws an error' do
-        expect { validate_storages_paths }.to raise_error('bar is a nested path of foo. Nested paths are not supported for repository storages. Please fix this in your gitlab.yml before starting GitLab.')
-      end
-    end
-
-    context 'with similar but un-nested storage paths' do
-      before do
-        mock_storages('foo' => Gitlab::GitalyClient::StorageSettings.new('path' => 'tmp/tests/paths/a/b/c'), 'bar' => Gitlab::GitalyClient::StorageSettings.new('path' => 'tmp/tests/paths/a/b/c2'))
-      end
-
-      it 'passes through' do
-        expect { validate_storages_paths }.not_to raise_error
-      end
-    end
-
-    describe 'inaccessible storage' do
-      before do
-        mock_storages('foo' => Gitlab::GitalyClient::StorageSettings.new('path' => 'tmp/tests/a/path/that/does/not/exist'))
-      end
-
-      it 'passes through with a warning' do
-        expect(Rails.logger).to receive(:error)
-        expect { validate_storages_paths }.not_to raise_error
-      end
-    end
-  end
-
   def mock_storages(storages)
     allow(Gitlab.config.repositories).to receive(:storages).and_return(storages)
   end