Add latest changes from gitlab-org/gitlab@master

CHANGELOG-EE.md

 Please view this file on the master branch, on stable branches it's out of date.
 
+## 12.9.0 (2020-03-22)
+
+### Removed (1 change)
+
+- Remove Security Dashboard button from project page. !24781
+
+### Fixed (20 changes)
+
+- Ensure User IDs are Removed When Moving to All Users Strategy. !25190
+- Fix dismissed vulnerability comment updates. !25328
+- Generate illustrations URLs in the backend. !25375
+- Check user role for each of namespaces while deciding if it is using a paid seat in it. !25437
+- Change MR approval popover to use hover trigger. !25494
+- Change copy of Welcome screen. !25526
+- Fix weight not showing in linked issues. !25718
+- Scope name uniqueness of approval rules to type. !25875
+- Pass correct Time object to Value Stream Management DataCollector. !25885
+- API for Namespace members has correct boolean if user using paid seat. !25988
+- Set the default SSL cert path environment variables when invoking indexer. !26036
+- Add pagination to Instance Level Security Dashboard project selector. !26138
+- Exclude creation event from audit changes. !26140
+- Resolve Usage Ping timeouts on security_products_usage for large hosts. !26296
+- Fix SCIM API not creating new users. !26514
+- BulkIndexer change _routing -> routing to support ES 7. !26639
+- Ensure design events are correctly visible. !26708
+- Add support for async search within Epics dropdown. !26980
+- Fix configuration issue for License Compliance. !27016
+- Geo - Fix ambiguous reference column while loading migrated local files to clean up. !27252
+
+### Changed (22 changes, 1 of them is from the community)
+
+- Add auto_renew attribute to gitlab_subscriptions and expose via api. !24896 (jejacks0n)
+- Remove admin user count from Users statistics. !25337
+- Improve error messages in the Dependency List. !25369
+- Do not automatically add approvers as participants of each merge request. !25546
+- Improve loading UX in the License Management list. !25620
+- Improve audit log & events header layout. !25821
+- Adjust the MR approval rule to use a smaller modal. !25955
+- Resolve Code review start time doesnt account for approvals. !25997
+- Prevent 'Invite group' for groups outside a group-managed account group. !26081
+- Improve language when MR Security Report is outdated. !26150
+- Prevent projects from being shared outside a group with managed accounts. !26163
+- Prevent projects from being shared outside a group with managed accounts for forked projects. !26186
+- Extract time range information to be reused. !26431
+- Restrict inviting outside group to a project forked from a group with enforced SSO. !26456
+- Add severity badge to security reports. !26715
+- Consider MR reassign as code review start. !26891
+- Add application limits for Ci::PipelineSchedule. !27004
+- Update license compliance policy classifications text in the UI. !27058
+- Remove approval metric recalculation when approval was removed. !27071
+- Move Value Stream Analytics to the group level. !27304
+- Remove release tag filter from Issues Analytics. !208903
+- Change dependency & license scan pipeline text message.
+
+### Performance (4 changes)
+
+- Optimize service desk issue counters query performance in usage data. !26445
+- Preload database records in bulk indexing. !26754
+- Geo: Make the LFS registry the SSOT to optimize query performance. !27154
+- Remove feature flag elastic_bulk_incremental_updates. !27293
+
+### Added (58 changes, 9 of them are from the community)
+
+- Audit number of required approvals changes. !7531
+- Add weight and progress information in Roadmap Epic bars. !18957
+- Adds the ability to dismiss multiple vulnerabilities. !21480
+- Add code quality tab to pipeline view. !21527
+- Add milestones to roadmap. !22748
+- Add Threat Monitoring page. !22911
+- Add epic in filtered search. !22958
+- Add deep links for cycle analytics. !23493
+- Allow existing users to transfer their account to SAML Group managed accounts. !24329
+- Use snowplow for user error reporting in trial. !24430
+- Time period for usage activity by stage in usage data. !24431
+- Adds sorting to group packages api. !24432
+- Export Board Milestones in Group Export. !24606
+- Add possibility to search for epics by partial iids using GraphQL. !24673
+- Add restrict outer forks functionality for group SAML. !24698
+- Release Jira connect feature. !24818
+- Adds project_path sorting to group package API. !24833
+- Add webhook limits for groups. !25129
+- Default design images to 100% of viewport width. !25327
+- Add npm_package_requests_forwarding to application settings. !25346
+- Add bot count to Users statistics in Admin Area. !25355
+- Adding CODEOWNER approval metrics for merge requests. !25386
+- Separate 4 ee entities into own class file. !25400 (Rajendra Kadam)
+- Improve zoom reset after resizing window in Design Management. !25406
+- Introduce Project level issues analytics. !25417
+- Add tooltip to guests count in Users Statistics. !25433
+- Separate epic, epic_issue, linked epic ee entities into own class file. !25592 (Rajendra Kadam)
+- Add API endpoints for Group Activity Analytics. !25606
+- Add GraphQL mutation to set the epic of an issue. !25628
+- Add merge request metrics first approved at timestamp. !25681
+- Separate ee entities into own class file. !25709 (Rajendra Kadam)
+- Embed cluster health metrics in GitLab-flavored Markdown. !25739
+- Always show the Design Tab on Issues but indicate user if feature is not available. !25796
+- Status page settings. !25820
+- Audit user blocked via API. !25872
+- Create a vulnerability-list component. !25927
+- Audit project MR approval permission changes. !25959
+- Add health status data to the epic GraphQL endpoint. !25989
+- Display correct approval rules based on target branch in Edit MR form. !26053
+- Activate triggering webhooks from a subgroup feature. !26070
+- Separate vulnerabilities, feature flags, dependency entities into own class files. !26088 (Rajendra Kadam)
+- Support drag-and-drop design uploads in Design Management. !26139
+- Add distinct user count of secure stage items to usage ping. !26194
+- Support drag-and-drop on existing designs in Design Management. !26446
+- Add worker to sync paid seats info daily in the background. !26467
+- Separate project alias, build info, package entities into own class files. !26493 (Rajendra Kadam)
+- Support async loading & search of projects within Epics Tree. !26661
+- Add code review reassign metric structure. !26664
+- Separate npm, subs and geo entities into own class file. !26674 (Rajendra Kadam)
+- Separate geo_node and license entities into own class file. !26685 (Rajendra Kadam)
+- Separate merge request approval entities into own class files. !26860 (Rajendra Kadam)
+- Include snippet description as part of snippet title search (advanced search, Elasticsearch reindexing required). !26870
+- Add Roadmap GraphQL and unfiltered epic aggregates. !26919
+- Add NPM package requests forwarding to npmjs.org. !26932
+- Separate approval entities into own class files. !27336 (Rajendra Kadam)
+- View description changes history of issues, epics and merge requests. !27445
+- Enable license compliance policies tab for on prem installs.
+- Add a bulk processor for ES incremental updates.
+
+### Other (5 changes, 1 of them is from the community)
+
+- Refactor WeightSelect to use gitlab-ui dropdown. !20770
+- Update to new severity badge design. !25489
+- Rename close_{at,by} to dismissed_{at,by} in Vulnerabilities. !25609 (Michał Zając)
+- Style MR approval popover link to be consistent text size. !25657
+- Allow users to be marked as service users. !202680
+
+
 ## 12.8.7 (2020-03-16)
 
 ### Fixed (1 change)

app/assets/javascripts/repository/components/preview/index.vue

@@ -56,7 +56,7 @@ export default {
         </gl-link>
       </div>
     </div>
-    <div class="blob-viewer">
+    <div class="blob-viewer" data-qa-selector="blob_viewer_content">
       <gl-loading-icon v-if="loading > 0" size="md" color="dark" class="my-4 mx-auto" />
       <div v-else-if="readme" ref="readme" v-html="readme.html"></div>
     </div>

app/models/merge_request_diff.rb

@@ -434,8 +434,50 @@ class MergeRequestDiff < ApplicationRecord
     merge_request_diff_files.reset
   end
 
+  # Transactionally migrate the current merge_request_diff_files entries from
+  # external storage, back to the database. This is the rollback operation for
+  # +migrate_files_to_external_storage!+
+  #
+  # If this diff isn't in external storage, the method is a no-op.
+  def migrate_files_to_database!
+    return unless stored_externally?
+    return if merge_request_diff_files.count == 0
+
+    rows = convert_external_diffs_to_database
+
+    transaction do
+      MergeRequestDiffFile.where(merge_request_diff_id: id).delete_all
+      Gitlab::Database.bulk_insert('merge_request_diff_files', rows)
+      update!(stored_externally: false)
+    end
+
+    # Only delete the external diff file after the contents have been saved to
+    # the database
+    remove_external_diff!
+    merge_request_diff_files.reset
+  end
+
   private
 
+  def convert_external_diffs_to_database
+    opening_external_diff do |external_file|
+      merge_request_diff_files.map do |diff_file|
+        row = diff_file.attributes.except('diff')
+
+        raise "Diff file lacks external diff offset or size: #{row.inspect}" unless
+          row['external_diff_offset'] && row['external_diff_size']
+
+        # The diff in the external file is already base64-encoded if necessary,
+        # matching the 'binary' attribute of the row. Reading it directly allows
+        # a cycle of decode-encode to be skipped
+        external_file.seek(row.delete('external_diff_offset'))
+        row['diff'] = external_file.read(row.delete('external_diff_size'))
+
+        row
+      end
+    end
+  end
+
   def diffs_in_batch_collection(batch_page, batch_size, diff_options:)
     Gitlab::Diff::FileCollection::MergeRequestDiffBatch.new(self,
                                                             batch_page,

app/views/layouts/nav/_breadcrumbs.html.haml

@@ -7,7 +7,7 @@
       = button_tag class: 'toggle-mobile-nav', type: 'button' do
         %span.sr-only= _("Open sidebar")
         = icon ('bars')
-    .breadcrumbs-links.js-title-container
+    .breadcrumbs-links.js-title-container{ data: { qa_selector: 'breadcrumb_links_content' } }
       %ul.list-unstyled.breadcrumbs-list.js-breadcrumbs-list
         - unless hide_top_links
           = header_title

changelogs/unreleased/37995-allow-external-diffs-to-be-removed.yml

+---
+title: Allow external diff files to be removed
+merge_request: 27602
+author:
+type: changed

doc/integration/saml.md

@@ -67,6 +67,8 @@ in your SAML IdP:
    auto_link_saml_user: true
    ```
 
+1. Ensure that the SAML [`NameID`](../user/group/saml_sso/index.md#nameid) and email address are fixed for each user, as described in the section on [Security](#security). Otherwise, your users will be able to sign in as other authorized users.
+
 1. Add the provider configuration:
 
    For Omnibus package:
@@ -430,6 +432,8 @@ args: {
 }
 ```
 
+Make sure you read the [Security](#security) section before changing this value.
+
 ## Response signature validation (required)
 
 We require Identity Providers to sign SAML responses to ensure that the assertions are
@@ -544,6 +548,17 @@ args: {
 
 GitLab will sign the request with the provided private key. GitLab will include the configured public x500 certificate in the metadata for your Identity Provider to validate the signature of the received request with. For more information on this option, see the [Ruby SAML gem documentation](https://github.com/onelogin/ruby-saml/tree/v1.7.0). The Ruby SAML gem is used by the [OmniAuth SAML gem](https://github.com/omniauth/omniauth-saml) to implement the client side of the SAML authentication.
 
+## Security
+
+Avoid user control of the following attributes:
+
+- [`*NameID*`](../user/group/saml_sso/index.md#nameid)
+- *Email* when used with `omniauth_auto_link_saml_user`
+
+These attributes define the SAML user. If users can change these attributes, they can impersonate others.
+
+Refer to the documentation for your [SAML Identity Provider](../user/group/saml_sso/index.md#providers) for information on how to fix these attributes.
+
 ## Troubleshooting
 
 ### GitLab+SAML Testing Environments

lib/gitlab/git_access_snippet.rb

@@ -98,7 +98,10 @@ module Gitlab
 
     def check_single_change_access(change)
       Checks::SnippetCheck.new(change, logger: logger).validate!
+
+      if Feature.enabled?(:snippet_count_check)
         Checks::PushFileCountCheck.new(change, repository: repository, limit: Snippet::MAX_FILE_COUNT, logger: logger).validate!
+      end
     rescue Checks::TimedLogger::TimeoutError
       raise TimeoutError, logger.full_message
     end

lib/gitlab/sidekiq_cluster.rb

 # frozen_string_literal: true
 
+require 'shellwords'
+
 module Gitlab
   module SidekiqCluster
     # The signals that should terminate both the master and workers.
@@ -73,9 +75,9 @@ module Gitlab
       counts = count_by_queue(queues)
 
       cmd = %w[bundle exec sidekiq]
-      cmd << "-c #{self.concurrency(queues, min_concurrency, max_concurrency)}"
+      cmd << "-c#{self.concurrency(queues, min_concurrency, max_concurrency)}"
       cmd << "-e#{env}"
-      cmd << "-gqueues: #{proc_details(counts)}"
+      cmd << "-gqueues:#{proc_details(counts)}"
       cmd << "-r#{directory}"
 
       counts.each do |queue, count|
@@ -83,7 +85,7 @@ module Gitlab
       end
 
       if dryrun
-        puts "Sidekiq command: #{cmd}" # rubocop:disable Rails/Output
+        puts Shellwords.join(cmd) # rubocop:disable Rails/Output
         return
       end
 
@@ -112,7 +114,7 @@ module Gitlab
         else
           "#{queue} (#{count})"
         end
-      end.join(', ')
+      end.join(',')
     end
 
     def self.concurrency(queues, min_concurrency, max_concurrency)

lib/gitlab/sidekiq_cluster/cli.rb

@@ -64,7 +64,9 @@ module Gitlab
             'No queues found, you must select at least one queue'
         end
 
+        unless @dryrun
           @logger.info("Starting cluster with #{queue_groups.length} processes")
+        end
 
         @processes = SidekiqCluster.start(
           queue_groups,

qa/qa.rb

@@ -385,6 +385,7 @@ module QA
     # Classes describing components that are used by several pages.
     #
     module Component
+      autoload :Breadcrumbs, 'qa/page/component/breadcrumbs'
       autoload :CiBadgeLink, 'qa/page/component/ci_badge_link'
       autoload :ClonePanel, 'qa/page/component/clone_panel'
       autoload :LazyLoader, 'qa/page/component/lazy_loader'

qa/qa/page/component/breadcrumbs.rb

+# frozen_string_literal: true
+
+module QA
+  module Page
+    module Component
+      module Breadcrumbs
+        def self.included(base)
+          base.view 'app/views/layouts/nav/_breadcrumbs.html.haml' do
+            element :breadcrumb_links_content
+          end
+        end
+
+        def has_breadcrumb?(text)
+          has_element?(:breadcrumb_links_content, text: text)
+        end
+      end
+    end
+  end
+end

qa/qa/page/project/show.rb

@@ -5,8 +5,13 @@ module QA
     module Project
       class Show < Page::Base
         include Page::Component::ClonePanel
+        include Page::Component::Breadcrumbs
         include Page::Project::SubMenus::Settings
 
+        view 'app/assets/javascripts/repository/components/preview/index.vue' do
+          element :blob_viewer_content
+        end
+
         view 'app/assets/javascripts/repository/components/table/row.vue' do
           element :file_name_link
         end
@@ -90,7 +95,7 @@ module QA
 
         def click_file(filename)
           within_element(:file_tree_table) do
-            click_on filename
+            click_element(:file_name_link, text: filename)
           end
         end
 
@@ -115,6 +120,10 @@ module QA
           has_element?(:project_name_content, text: name)
         end
 
+        def has_readme_content?(text)
+          has_element?(:blob_viewer_content, text: text)
+        end
+
         def last_commit_content
           find_element(:commit_content).text
         end

qa/qa/specs/features/browser_ui/1_manage/group/transfer_project_spec.rb

@@ -47,8 +47,8 @@ module QA
         Page::Project::Settings::Main.perform(&:click_project)
 
         Page::Project::Show.perform do |project|
-          expect(project).to have_text(target_group.path)
-          expect(project).to have_text(edited_readme_content)
+          expect(project).to have_breadcrumb(target_group.path)
+          expect(project).to have_readme_content(edited_readme_content)
         end
       end
     end

qa/qa/specs/features/browser_ui/3_create/repository/protocol_v2_push_ssh_spec.rb

@@ -60,12 +60,15 @@ module QA
           end
         end
 
-        project.visit!
         project.wait_for_push_new_branch
+        project.visit!
 
-        expect(page).to have_content(file_name)
-        expect(page).to have_content(file_content)
         expect(git_protocol_reported).to eq(git_protocol)
+
+        Page::Project::Show.perform do |show|
+          expect(show).to have_file(file_name)
+          expect(show).to have_readme_content(file_content)
+        end
       end
     end
   end

qa/qa/specs/features/browser_ui/3_create/repository/push_http_private_token_spec.rb

@@ -24,8 +24,10 @@ module QA
 
         Page::Project::Show.perform(&:wait_for_viewers_to_load)
 
-        expect(page).to have_content('README.md')
-        expect(page).to have_content('This is a test project')
+        Page::Project::Show.perform do |project|
+          expect(project).to have_file('README.md')
+          expect(project).to have_readme_content('This is a test project')
+        end
       end
     end
   end

qa/qa/specs/features/browser_ui/3_create/repository/push_over_http_spec.rb

@@ -12,8 +12,10 @@ module QA
           push.commit_message = 'Add README.md'
         end.project.visit!
 
-        expect(page).to have_content('README.md')
-        expect(page).to have_content('This is a test project')
+        Page::Project::Show.perform do |project|
+          expect(project).to have_file('README.md')
+          expect(project).to have_readme_content('This is a test project')
+        end
       end
     end
   end

spec/bin/sidekiq_cluster_spec.rb

 # frozen_string_literal: true
 
 require 'spec_helper'
+require 'shellwords'
 
 describe 'bin/sidekiq-cluster' do
   using RSpec::Parameterized::TableSyntax
@@ -18,9 +19,9 @@ describe 'bin/sidekiq-cluster' do
         output, status = Gitlab::Popen.popen(cmd, Rails.root.to_s)
 
         expect(status).to be(0)
-        expect(output).to include('"bundle", "exec", "sidekiq"')
-        expect(output).to include(included)
-        expect(output).not_to include(excluded)
+        expect(output).to include('bundle exec sidekiq')
+        expect(Shellwords.split(output)).to include(included)
+        expect(Shellwords.split(output)).not_to include(excluded)
       end
     end
   end
@@ -36,9 +37,9 @@ describe 'bin/sidekiq-cluster' do
         output, status = Gitlab::Popen.popen(cmd, Rails.root.to_s)
 
         expect(status).to be(0)
-        expect(output).to include('"bundle", "exec", "sidekiq"')
-        expect(output).to include('-qdefault,1')
-        expect(output).to include('-qcronjob:ci_archive_traces_cron,1')
+        expect(output).to include('bundle exec sidekiq')
+        expect(Shellwords.split(output)).to include('-qdefault,1')
+        expect(Shellwords.split(output)).to include('-qcronjob:ci_archive_traces_cron,1')
       end
     end
   end

spec/frontend/repository/components/preview/__snapshots__/index_spec.js.snap

@@ -27,6 +27,7 @@ exports[`Repository file preview component renders file HTML 1`] = `
    
   <div
     class="blob-viewer"
+    data-qa-selector="blob_viewer_content"
   >
     <div>
       <div

spec/lib/gitlab/git_access_snippet_spec.rb

@@ -209,6 +209,16 @@ describe Gitlab::GitAccessSnippet do
 
       expect { push_access_check }.to raise_forbidden('foo')
     end
+
+    context 'when feature flag :snippet_count_check is disabled' do
+      it 'does not check push file count' do
+        stub_feature_flags(snippet_count_check: false)
+
+        expect(Gitlab::Checks::PushFileCountCheck).not_to receive(:new)
+
+        expect { push_access_check }.not_to raise_error
+      end
+    end
   end
 
   private

spec/models/merge_request_diff_spec.rb

@@ -156,6 +156,41 @@ describe MergeRequestDiff do
     end
   end
 
+  describe '#migrate_files_to_database!' do
+    let(:diff) { create(:merge_request).merge_request_diff }
+
+    it 'converts from external to in-database storage' do
+      stub_external_diffs_setting(enabled: true)
+
+      expect(diff).to be_stored_externally
+      expect(diff).to receive(:update!).and_call_original
+
+      file = diff.external_diff
+      file_data = file.read
+      diff.migrate_files_to_database!
+
+      expect(diff).not_to be_stored_externally
+      expect(file).not_to exist
+      expect(diff.merge_request_diff_files.map(&:diff).join('')).to eq(file_data)
+    end
+
+    it 'does nothing with an in-database diff' do
+      expect(diff).not_to be_stored_externally
+      expect(diff).not_to receive(:update!)
+
+      diff.migrate_files_to_database!
+    end
+
+    it 'does nothing with an empty diff' do
+      stub_external_diffs_setting(enabled: true)
+      MergeRequestDiffFile.where(merge_request_diff_id: diff.id).delete_all
+
+      expect(diff).not_to receive(:update!)
+
+      diff.migrate_files_to_database!
+    end
+  end
+
   describe '#latest?' do
     let!(:mr) { create(:merge_request, :with_diffs) }
     let!(:first_diff) { mr.merge_request_diff }