Merge branch 'feature/custom-highlighting' into 'master'

Add custom highlighting via .gitattributes

## What does this MR do?
Allows user control of language selection via a `gitlab-language` entry in `.gitattributes`

## Are there points in the code the reviewer needs to double check?
(paired with @stanhu)

## Why was this MR needed?
Guessing languages by filename is fraught and often wrong. In one project, `foo.pl` may be perl, and in another it may be prolog. Users might have a Thingfile that needs ruby highlighting, or depend on things that can't work in general, like `*.C` (capitalized) mapping to C++ instead of C.

This allows the user to override language choice so they never have to look at a mis-highlighted file.

## What are the relevant issue numbers?
https://github.com/jneen/rouge/issues/494
https://gitlab.com/gitlab-org/gitlab-ce/issues/13818 (*.tpl can't in general map to Smarty)
https://gitlab.com/gitlab-org/gitlab-ce/issues/13615 (in cases we don't have a language and mis-identify it, users could map to 'text' to turn off highlighting)

## Screenshots (if relevant)

## Does this MR meet the acceptance criteria?

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [x] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [x] API support added (N/A)
- [x] Tests
  - [x] Added for this feature/bug
  - [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

See merge request !4606

app/helpers/blob_helper.rb

 module BlobHelper
-  def highlighter(blob_name, blob_content, nowrap: false)
-    Gitlab::Highlight.new(blob_name, blob_content, nowrap: nowrap)
+  def highlighter(blob_name, blob_content, repository: nil, nowrap: false)
+    Gitlab::Highlight.new(blob_name, blob_content, nowrap: nowrap, repository: repository)
   end
 
-  def highlight(blob_name, blob_content, nowrap: false, plain: false)
-    Gitlab::Highlight.highlight(blob_name, blob_content, nowrap: nowrap, plain: plain)
+  def highlight(blob_name, blob_content, repository: nil, nowrap: false, plain: false)
+    Gitlab::Highlight.highlight(blob_name, blob_content, nowrap: nowrap, plain: plain, repository: repository)
   end
 
   def no_highlight_files

app/models/repository.rb

@@ -978,6 +978,10 @@ class Repository
     raw_repository.ls_files(actual_ref)
   end
 
+  def gitattribute(path, name)
+    raw_repository.attributes(path)[name]
+  end
+
   def copy_gitattributes(ref)
     actual_ref = ref || root_ref
     begin

app/models/snippet.rb

@@ -20,6 +20,7 @@ class Snippet < ActiveRecord::Base
     length: { within: 0..255 },
     format: { with: Gitlab::Regex.file_name_regex,
               message: Gitlab::Regex.file_name_regex_message }
+
   validates :content, presence: true
   validates :visibility_level, inclusion: { in: Gitlab::VisibilityLevel.values }
 
@@ -81,6 +82,11 @@ class Snippet < ActiveRecord::Base
     0
   end
 
+  # alias for compatibility with blobs and highlighting
+  def path
+    file_name
+  end
+
   def name
     file_name
   end

app/views/projects/blob/_text.html.haml

@@ -16,4 +16,4 @@
       .file-content.code
         .nothing-here-block Empty file
     - else
-      = render 'shared/file_highlight', blob: blob
+      = render 'shared/file_highlight', blob: blob, repository: @repository

app/views/shared/_file_highlight.html.haml

+- repository = nil unless local_assigns.key?(:repository)
+
 .file-content.code.js-syntax-highlight
   .line-numbers
     - if blob.data.present?
@@ -11,4 +13,4 @@
           = link_icon
           = i
   .blob-content{data: {blob_id: blob.id}}
-    = highlight(blob.name, blob.data, plain: blob.no_highlighting?)
+    = highlight(blob.path, blob.data, repository: repository, plain: blob.no_highlighting?)

doc/user/project/highlighting.md

+[Rouge]: https://rubygems.org/gems/rouge
+
+# Syntax Highlighting
+
+GitLab provides syntax highlighting on all files and snippets through the [Rouge][] rubygem. It will try to guess what language to use based on the file extension, which most of the time is sufficient.
+
+If GitLab is guessing wrong, you can override its choice of language using the `gitlab-language` attribute in `.gitattributes`. For example, if you are working in a Prolog project and using the `.pl` file extension (which would normally be highlighted as Perl), you can add the following to your `.gitattributes` file:
+
+``` conf
+*.pl gitlab-language=prolog
+```
+
+When you check in and push that change, all `*.pl` files in your project will be highlighted as Prolog.
+
+The paths here are simply git's builtin [`.gitattributes` interface](https://git-scm.com/docs/gitattributes).  So, if you were to invent a file format called a `Nicefile` at the root of your project that used ruby syntax, all you need is:
+
+``` conf
+/Nicefile gitlab-language=ruby
+```
+
+To disable highlighting entirely, use `gitlab-language=text`. Lots more fun shenanigans are available through CGI options, such as:
+
+``` conf
+# json with erb in it
+/my-cool-file gitlab-language=erb?parent=json
+
+# an entire file of highlighting errors!
+/other-file gitlab-language=text?token=Error
+```
+
+Please note that these configurations will only take effect when the `.gitattributes` file is in your default branch (usually `master`).

lib/gitlab/blame.rb

@@ -41,7 +41,8 @@ module Gitlab
 
     def highlighted_lines
       @blob.load_all_data!(repository)
-      @highlighted_lines ||= Gitlab::Highlight.highlight(@blob.name, @blob.data).lines
+      @highlighted_lines ||=
+        Gitlab::Highlight.highlight(@blob.path, @blob.data, repository: repository).lines
     end
 
     def project

lib/gitlab/highlight.rb

 module Gitlab
   class Highlight
-    def self.highlight(blob_name, blob_content, nowrap: true, plain: false)
-      new(blob_name, blob_content, nowrap: nowrap).
+    def self.highlight(blob_name, blob_content, repository: nil, nowrap: true, plain: false)
+      new(blob_name, blob_content, nowrap: nowrap, repository: repository).
         highlight(blob_content, continue: false, plain: plain)
     end
 
@@ -10,12 +10,21 @@ module Gitlab
       return [] unless blob
 
       blob.load_all_data!(repository)
-      highlight(file_name, blob.data).lines.map!(&:html_safe)
+      highlight(file_name, blob.data, repository: repository).lines.map!(&:html_safe)
     end
 
-    def initialize(blob_name, blob_content, nowrap: true)
+    attr_reader :lexer
+    def initialize(blob_name, blob_content, repository: nil, nowrap: true)
+      @blob_name = blob_name
+      @blob_content = blob_content
+      @repository = repository
       @formatter = rouge_formatter(nowrap: nowrap)
-      @lexer = Rouge::Lexer.guess(filename: blob_name, source: blob_content).new rescue Rouge::Lexers::PlainText
+
+      @lexer = custom_language || begin
+        Rouge::Lexer.guess(filename: blob_name, source: blob_content).new
+      rescue Rouge::Lexer::AmbiguousGuess => e
+        e.alternatives.sort_by(&:tag).first
+      end
     end
 
     def highlight(text, continue: true, plain: false)
@@ -30,6 +39,14 @@ module Gitlab
 
     private
 
+    def custom_language
+      language_name = @repository && @repository.gitattribute(@blob_name, 'gitlab-language')
+
+      return nil unless language_name
+
+      Rouge::Lexer.find_fancy(language_name)
+    end
+
     def rouge_formatter(options = {})
       options = options.reverse_merge(
         nowrap: true,

spec/lib/gitlab/highlight_spec.rb

@@ -4,6 +4,7 @@ describe Gitlab::Highlight, lib: true do
   include RepoHelpers
 
   let(:project) { create(:project) }
+  let(:repository) { project.repository }
   let(:commit) { project.commit(sample_commit.id) }
 
   describe '.highlight_lines' do
@@ -18,4 +19,30 @@ describe Gitlab::Highlight, lib: true do
     end
   end
 
+  describe 'custom highlighting from .gitattributes' do
+    let(:branch) { 'gitattributes' }
+    let(:blob) { repository.blob_at_branch(branch, path) }
+
+    let(:highlighter) do
+      Gitlab::Highlight.new(blob.path, blob.data, repository: repository)
+    end
+
+    before { project.change_head('gitattributes') }
+
+    describe 'basic language selection' do
+      let(:path) { 'custom-highlighting/test.gitlab-custom' }
+      it 'highlights as ruby' do
+        expect(highlighter.lexer.tag).to eq 'ruby'
+      end
+    end
+
+    describe 'cgi options' do
+      let(:path) { 'custom-highlighting/test.gitlab-cgi' }
+
+      it 'highlights as json with erb' do
+        expect(highlighter.lexer.tag).to eq 'erb'
+        expect(highlighter.lexer.parent.tag).to eq 'json'
+      end
+    end
+  end
 end

spec/support/test_env.rb

@@ -17,6 +17,7 @@ module TestEnv
     "'test'"           => 'e56497b',
     'orphaned-branch'  => '45127a9',
     'binary-encoding'  => '7b1cf43',
+    'gitattributes'    => '5a62481',
   }
 
   # gitlab-test-fork is a fork of gitlab-fork, but we don't necessarily