Merge branch 'ericidema/gitlab-ce-import-with-github-personal-access-tokens' into 'master'

Allow importing from Github using Personal Access Tokens

_Originally opened at !4005 by @ericidema._

------

## What does this MR do?

* Made changes to `Gitlab::GithubImport::Client` so that it can be used with Github Personal Access Tokens without the need for OAuth.
* Added UI to collect Personal Access Token from user.
* Detect if the user has logged in with GitHub and use OAuth to skip the Personal Access Token form.

## Are there points in the code the reviewer needs to double check?

Twin Omnibus MR: https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/774

## What are the relevant issue numbers?

Closes #13970.

## Screenshots

### GitHub import is configured

![github_import_configured](/uploads/151e4f0edf3f87bfa03c2d97dda8b3d8/github_import_configured.png)

-----

### GitHub import is not configured

![github_import_not_configured](/uploads/cb129f7e2ffe66cceb28ccd9af480284/github_import_not_configured.png)

-----

## Does this MR meet the acceptance criteria?

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [x] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- Tests
  - [x] Added for this feature/bug
  - [ ] All builds are passing
- [ ] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

See merge request !4938

CHANGELOG

@@ -18,6 +18,7 @@ v 8.10.0 (unreleased)
   - Exclude email check from the standard health check
   - Fix changing issue state columns in milestone view
   - Add notification settings dropdown for groups
+  - Allow importing from Github using Personal Access Tokens. (Eric K Idema)
   - Fix user creation with stronger minimum password requirements !4054 (nathan-pmt)
   - PipelinesFinder uses git cache data
   - Check for conflicts with existing Project's wiki path when creating a new project.

app/controllers/import/github_controller.rb

 class Import::GithubController < Import::BaseController
   before_action :verify_github_import_enabled
-  before_action :github_auth, except: :callback
+  before_action :github_auth, only: [:status, :jobs, :create]
 
   rescue_from Octokit::Unauthorized, with: :github_unauthorized
 
+  helper_method :logged_in_with_github?
+
+  def new
+    if logged_in_with_github?
+      go_to_github_for_permissions
+    elsif session[:github_access_token]
+      redirect_to status_import_github_url
+    end
+  end
+
   def callback
     session[:github_access_token] = client.get_token(params[:code])
     redirect_to status_import_github_url
   end
 
+  def personal_access_token
+    session[:github_access_token] = params[:personal_access_token]
+    redirect_to status_import_github_url
+  end
+
   def status
     @repos = client.repos
     @already_added_projects = current_user.created_projects.where(import_type: "github")
@@ -57,10 +72,14 @@ class Import::GithubController < Import::BaseController
   end
 
   def github_unauthorized
-    go_to_github_for_permissions
+    session[:github_access_token] = nil
+    redirect_to new_import_github_url,
+      alert: 'Access denied to your GitHub account.'
   end
 
-  private
+  def logged_in_with_github?
+    current_user.identities.exists?(provider: 'github')
+  end
 
   def access_params
     { github_access_token: session[:github_access_token] }

app/views/import/github/new.html.haml

+- page_title "GitHub Import"
+- header_title "Projects", root_path
+
+%h3.page-title
+  = icon 'github', text: 'Import Projects from GitHub'
+
+- if github_import_configured?
+  %p
+    To import a GitHub project, you first need to authorize GitLab to access
+    the list of your GitHub repositories:
+
+  = link_to 'List Your GitHub Repositories', status_import_github_path, class: 'btn btn-success'
+
+  %hr
+
+%p
+  - if github_import_configured?
+    Alternatively,
+  - else
+    To import a GitHub project,
+  you can use a
+  = succeed '.' do
+    = link_to 'Personal Access Token', 'https://github.com/settings/tokens'
+  When you create your Personal Access Token,
+  you will need to select the <code>repo</code> scope, so we can display a
+  list of your public and private repositories which are available for import.
+
+= form_tag personal_access_token_import_github_path, method: :post, class: 'form-inline' do
+  .form-group
+    = text_field_tag :personal_access_token, '', class: 'form-control', placeholder: "Personal Access Token", size: 40
+    = submit_tag 'List Your GitHub Repositories', class: 'btn btn-success'
+
+- unless github_import_configured?
+  %hr
+  %p
+    Note:
+    - if current_user.admin?
+      As an administrator you may like to configure
+    - else
+      Consider asking your GitLab administrator to configure
+    = link_to 'GitHub integration', help_page_path("integration", "github")
+    which will allow login via GitHub and allow importing projects without
+    generating a Personal Access Token.

app/views/projects/_github_import_modal.html.haml

-%div#github_import_modal.modal
-  .modal-dialog
-    .modal-content
-      .modal-header
-        %a.close{href: "#", "data-dismiss" => "modal"} ×
-        %h3 Import projects from GitHub
-      .modal-body
-        To enable importing projects from GitHub,
-        - if current_user.admin?
-          as administrator you need to configure
-        - else
-          ask your Gitlab administrator to configure
-        == #{link_to 'OAuth integration', help_page_path("integration", "github")}.

app/views/projects/new.html.haml

@@ -23,6 +23,7 @@
                   .input-group-addon
                     = root_url
                   = f.select :namespace_id, namespaces_options(params[:namespace_id] || :current_user, display_path: true), {}, {class: 'select2 js-select-namespace', tabindex: 1}
+
                 - else
                   .input-group-addon.static-namespace
                     #{root_url}#{current_user.username}/
@@ -44,15 +45,8 @@
               .col-sm-12.import-buttons
                 %div
                   - if github_import_enabled?
-                    - if github_import_configured?
-                      = link_to status_import_github_path, class: 'btn import_github' do
-                        %i.fa.fa-github
-                        GitHub
-                    - else
-                      = link_to '#', class: 'how_to_import_link btn import_github' do
-                        %i.fa.fa-github
-                        GitHub
-                      = render 'github_import_modal'
+                    = link_to new_import_github_path, class: 'btn import_github' do
+                      = icon 'github', text: 'GitHub'
                 %div
                   - if bitbucket_import_enabled?
                     - if bitbucket_import_configured?

config/initializers/rack_attack.rb.example

@@ -10,7 +10,8 @@ paths_to_be_protected = [
   "#{Rails.application.config.relative_url_root}/api/#{API::API.version}/session",
   "#{Rails.application.config.relative_url_root}/users",
   "#{Rails.application.config.relative_url_root}/users/confirmation",
-  "#{Rails.application.config.relative_url_root}/unsubscribes/"
+  "#{Rails.application.config.relative_url_root}/unsubscribes/",
+  "#{Rails.application.config.relative_url_root}/import/github/personal_access_token"
 
 ]
 

config/routes.rb

@@ -139,6 +139,7 @@ Rails.application.routes.draw do
   #
   namespace :import do
     resource :github, only: [:create, :new], controller: :github do
+      post :personal_access_token
       get :status
       get :callback
       get :jobs

doc/workflow/importing/import_projects_from_github.md

 # Import your project from GitHub to GitLab
 
 >**Note:**
-In order to enable the GitHub import setting, you should first
-enable the [GitHub integration][gh-import] in your GitLab instance.
+In order to enable the GitHub import setting, you may also want to
+enable the [GitHub integration][gh-import] in your GitLab instance. This
+configuration is optional, you will be able import your GitHub repositories
+with a Personal Access Token.
 
 At its current state, GitHub importer can import:
 
@@ -20,9 +22,15 @@ It is not yet possible to import your cross-repository pull requests (those from
 forks). We are working on improving this in the near future.
 
 The importer page is visible when you [create a new project][new-project].
-Click on the **GitHub** link and you will be redirected to GitHub for
-permission to access your projects. After accepting, you'll be automatically
-redirected to the importer.
+Click on the **GitHub** link and, if you are logged in via the GitHub
+integration, you will be redirected to GitHub for permission to access your
+projects. After accepting, you'll be automatically redirected to the importer.
+
+If you are not using the GitHub integration, you can still perform a one-off
+authorization with GitHub to access your projects.
+
+Alternatively, you can also enter a GitHub Personal Access Token. Once you enter
+your token, you'll be taken to the importer.
 
 ![New project page on GitLab](img/import_projects_from_github_new_project_page.png)
 

features/dashboard/new_project.feature

@@ -21,7 +21,7 @@ Background:
   Scenario: I should see instructions on how to import from GitHub
   Given I see "New Project" page
   When I click on "Import project from GitHub"
-  Then I see instructions on how to import from GitHub
+  Then I am redirected to the GitHub import page
 
   @javascript
   Scenario: I should see Google Code import page

features/steps/dashboard/new_project.rb

@@ -28,14 +28,8 @@ class Spinach::Features::NewProject < Spinach::FeatureSteps
     first('.import_github').click
   end
 
-  step 'I see instructions on how to import from GitHub' do
-    github_modal = first('.modal-body')
-    expect(github_modal).to be_visible
-    expect(github_modal).to have_content "To enable importing projects from GitHub"
-
-    page.all('.modal-body').each do |element|
-      expect(element).not_to be_visible unless element == github_modal
-    end
+  step 'I am redirected to the GitHub import page' do
+    expect(current_path).to eq new_import_github_path
   end
 
   step 'I click on "Repo by URL"' do

lib/gitlab/github_import/client.rb

@@ -4,26 +4,39 @@ module Gitlab
       GITHUB_SAFE_REMAINING_REQUESTS = 100
       GITHUB_SAFE_SLEEP_TIME = 500
 
-      attr_reader :client, :api
+      attr_reader :access_token
 
       def initialize(access_token)
-        @client = ::OAuth2::Client.new(
-          config.app_id,
-          config.app_secret,
-          github_options.merge(ssl: { verify: config['verify_ssl'] })
-        )
+        @access_token = access_token
 
         if access_token
           ::Octokit.auto_paginate = false
+        end
+      end
+
+      def api
+        @api ||= ::Octokit::Client.new(
+          access_token: access_token,
+          api_endpoint: github_options[:site],
+          # If there is no config, we're connecting to github.com and we
+          # should verify ssl.
+          connection_options: {
+            ssl: { verify: config ? config['verify_ssl'] : true }
+          }
+        )
+      end
 
-          @api = ::Octokit::Client.new(
-            access_token: access_token,
-            api_endpoint: github_options[:site],
-            connection_options: {
-              ssl: { verify: config['verify_ssl'] }
-            }
-          )
+      def client
+        unless config
+          raise Projects::ImportService::Error,
+            'OAuth configuration for GitHub missing.'
         end
+
+        @client ||= ::OAuth2::Client.new(
+          config.app_id,
+          config.app_secret,
+          github_options.merge(ssl: { verify: config['verify_ssl'] })
+        )
       end
 
       def authorize_url(redirect_uri)
@@ -56,7 +69,11 @@ module Gitlab
       end
 
       def github_options
-        config["args"]["client_options"].deep_symbolize_keys
+        if config
+          config["args"]["client_options"].deep_symbolize_keys
+        else
+          OmniAuth::Strategies::GitHub.default_options[:client_options].symbolize_keys
+        end
       end
 
       def rate_limit

spec/controllers/import/github_controller_spec.rb

@@ -16,6 +16,24 @@ describe Import::GithubController do
     allow(controller).to receive(:github_import_enabled?).and_return(true)
   end
 
+  describe "GET new" do
+    it "redirects to GitHub for an access token if logged in with GitHub" do
+      allow(controller).to receive(:logged_in_with_github?).and_return(true)
+      expect(controller).to receive(:go_to_github_for_permissions)
+
+      get :new
+    end
+
+    it "redirects to status if we already have a token" do
+      assign_session_token
+      allow(controller).to receive(:logged_in_with_github?).and_return(false)
+
+      get :new
+
+      expect(controller).to redirect_to(status_import_github_url)
+    end
+  end
+
   describe "GET callback" do
     it "updates access token" do
       token = "asdasd12345"
@@ -32,6 +50,20 @@ describe Import::GithubController do
     end
   end
 
+  describe "POST personal_access_token" do
+    it "updates access token" do
+      token = "asdfasdf9876"
+
+      allow_any_instance_of(Gitlab::GithubImport::Client).
+        to receive(:user).and_return(true)
+
+      post :personal_access_token, personal_access_token: token
+
+      expect(session[:github_access_token]).to eq(token)
+      expect(controller).to redirect_to(status_import_github_url)
+    end
+  end
+
   describe "GET status" do
     before do
       @repo = OpenStruct.new(login: 'vim', full_name: 'asd/vim')
@@ -59,6 +91,17 @@ describe Import::GithubController do
       expect(assigns(:already_added_projects)).to eq([@project])
       expect(assigns(:repos)).to eq([])
     end
+
+    it "handles an invalid access token" do
+      allow_any_instance_of(Gitlab::GithubImport::Client).
+        to receive(:repos).and_raise(Octokit::Unauthorized)
+
+      get :status
+
+      expect(session[:github_access_token]).to eq(nil)
+      expect(controller).to redirect_to(new_import_github_url)
+      expect(flash[:alert]).to eq('Access denied to your GitHub account.')
+    end
   end
 
   describe "POST create" do

spec/lib/gitlab/github_import/client_spec.rb

@@ -20,6 +20,20 @@ describe Gitlab::GithubImport::Client, lib: true do
     expect { client.api }.not_to raise_error
   end
 
+  context 'when config is missing' do
+    before do
+      allow(Gitlab.config.omniauth).to receive(:providers).and_return([])
+    end
+
+    it 'is still possible to get an Octokit client' do
+      expect { client.api }.not_to raise_error
+    end
+
+    it 'is not be possible to get an OAuth2 client' do
+      expect { client.client }.to raise_error(Projects::ImportService::Error)
+    end
+  end
+
   context 'allow SSL verification to be configurable on API' do
     before do
       github_provider['verify_ssl'] = false