Commit 502cbda1 authored by Kamil Trzciński's avatar Kamil Trzciński

Merge branch 'ci-variable-expression-con-dis-junction' into 'master'

CI variable expression conjunction/disjunction

See merge request gitlab-org/gitlab-ce!27925
parents 8501edcd cfaac753
---
title: Add support for && and || to CI Pipeline Expressions. Change CI variable expression matching for Lexeme::Pattern to eagerly return tokens.
merge_request: 27925
author: Martin Manelli
type: added
...@@ -479,6 +479,7 @@ Below you can find supported syntax reference: ...@@ -479,6 +479,7 @@ Below you can find supported syntax reference:
1. Equality matching using a string 1. Equality matching using a string
> Example: `$VARIABLE == "some value"` > Example: `$VARIABLE == "some value"`
> Example: `$VARIABLE != "some value"` _(added in 11.11)_ > Example: `$VARIABLE != "some value"` _(added in 11.11)_
You can use equality operator `==` or `!=` to compare a variable content to a You can use equality operator `==` or `!=` to compare a variable content to a
...@@ -489,6 +490,7 @@ Below you can find supported syntax reference: ...@@ -489,6 +490,7 @@ Below you can find supported syntax reference:
1. Checking for an undefined value 1. Checking for an undefined value
> Example: `$VARIABLE == null` > Example: `$VARIABLE == null`
> Example: `$VARIABLE != null` _(added in 11.11)_ > Example: `$VARIABLE != null` _(added in 11.11)_
It sometimes happens that you want to check whether a variable is defined It sometimes happens that you want to check whether a variable is defined
...@@ -499,6 +501,7 @@ Below you can find supported syntax reference: ...@@ -499,6 +501,7 @@ Below you can find supported syntax reference:
1. Checking for an empty variable 1. Checking for an empty variable
> Example: `$VARIABLE == ""` > Example: `$VARIABLE == ""`
> Example: `$VARIABLE != ""` _(added in 11.11)_ > Example: `$VARIABLE != ""` _(added in 11.11)_
If you want to check whether a variable is defined, but is empty, you can If you want to check whether a variable is defined, but is empty, you can
...@@ -508,6 +511,7 @@ Below you can find supported syntax reference: ...@@ -508,6 +511,7 @@ Below you can find supported syntax reference:
1. Comparing two variables 1. Comparing two variables
> Example: `$VARIABLE_1 == $VARIABLE_2` > Example: `$VARIABLE_1 == $VARIABLE_2`
> Example: `$VARIABLE_1 != $VARIABLE_2` _(added in 11.11)_ > Example: `$VARIABLE_1 != $VARIABLE_2` _(added in 11.11)_
It is possible to compare two variables. This is going to compare values It is possible to compare two variables. This is going to compare values
...@@ -527,6 +531,7 @@ Below you can find supported syntax reference: ...@@ -527,6 +531,7 @@ Below you can find supported syntax reference:
1. Pattern matching _(added in 11.0)_ 1. Pattern matching _(added in 11.0)_
> Example: `$VARIABLE =~ /^content.*/` > Example: `$VARIABLE =~ /^content.*/`
> Example: `$VARIABLE_1 !~ /^content.*/` _(added in 11.11)_ > Example: `$VARIABLE_1 !~ /^content.*/` _(added in 11.11)_
It is possible perform pattern matching against a variable and regular It is possible perform pattern matching against a variable and regular
...@@ -536,6 +541,19 @@ Below you can find supported syntax reference: ...@@ -536,6 +541,19 @@ Below you can find supported syntax reference:
Pattern matching is case-sensitive by default. Use `i` flag modifier, like Pattern matching is case-sensitive by default. Use `i` flag modifier, like
`/pattern/i` to make a pattern case-insensitive. `/pattern/i` to make a pattern case-insensitive.
1. Conjunction / Disjunction
> Example: `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 == "something"`
> Example: `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 =~ /thing$/ && $VARIABLE3`
> Example: `$VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/ && $VARIABLE3`
It is possible to join multiple conditions using `&&` or `||`. Any of the otherwise
supported syntax may be used in a conjunctive or disjunctive statement.
Precedence of operators follows standard Ruby 2.5 operation
[precedence](https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html).
## Debug tracing ## Debug tracing
> Introduced in GitLab Runner 1.7. > Introduced in GitLab Runner 1.7.
......
# frozen_string_literal: true
module Gitlab
module Ci
module Pipeline
module Expression
module Lexeme
class And < Lexeme::Operator
PATTERN = /&&/.freeze
def evaluate(variables = {})
@left.evaluate(variables) && @right.evaluate(variables)
end
def self.build(_value, behind, ahead)
new(behind, ahead)
end
def self.precedence
11 # See: https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html
end
end
end
end
end
end
end
...@@ -15,10 +15,14 @@ module Gitlab ...@@ -15,10 +15,14 @@ module Gitlab
end end
def self.scan(scanner) def self.scan(scanner)
if scanner.scan(self::PATTERN) if scanner.scan(pattern)
Expression::Token.new(scanner.matched, self) Expression::Token.new(scanner.matched, self)
end end
end end
def self.pattern
self::PATTERN
end
end end
end end
end end
......
...@@ -8,11 +8,6 @@ module Gitlab ...@@ -8,11 +8,6 @@ module Gitlab
class Equals < Lexeme::Operator class Equals < Lexeme::Operator
PATTERN = /==/.freeze PATTERN = /==/.freeze
def initialize(left, right)
@left = left
@right = right
end
def evaluate(variables = {}) def evaluate(variables = {})
@left.evaluate(variables) == @right.evaluate(variables) @left.evaluate(variables) == @right.evaluate(variables)
end end
...@@ -20,6 +15,10 @@ module Gitlab ...@@ -20,6 +15,10 @@ module Gitlab
def self.build(_value, behind, ahead) def self.build(_value, behind, ahead)
new(behind, ahead) new(behind, ahead)
end end
def self.precedence
10 # See: https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html
end
end end
end end
end end
......
...@@ -8,21 +8,36 @@ module Gitlab ...@@ -8,21 +8,36 @@ module Gitlab
class Matches < Lexeme::Operator class Matches < Lexeme::Operator
PATTERN = /=~/.freeze PATTERN = /=~/.freeze
def initialize(left, right)
@left = left
@right = right
end
def evaluate(variables = {}) def evaluate(variables = {})
text = @left.evaluate(variables) text = @left.evaluate(variables)
regexp = @right.evaluate(variables) regexp = @right.evaluate(variables)
regexp.scan(text.to_s).any? regexp.scan(text.to_s).any?
if ci_variables_complex_expressions?
# return offset of first match, or nil if no matches
if match = regexp.scan(text.to_s).first
text.to_s.index(match)
end
else
# return true or false
regexp.scan(text.to_s).any?
end
end end
def self.build(_value, behind, ahead) def self.build(_value, behind, ahead)
new(behind, ahead) new(behind, ahead)
end end
def self.precedence
10 # See: https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html
end
private
def ci_variables_complex_expressions?
Feature.enabled?(:ci_variables_complex_expressions, default_enabled: true)
end
end end
end end
end end
......
...@@ -8,11 +8,6 @@ module Gitlab ...@@ -8,11 +8,6 @@ module Gitlab
class NotEquals < Lexeme::Operator class NotEquals < Lexeme::Operator
PATTERN = /!=/.freeze PATTERN = /!=/.freeze
def initialize(left, right)
@left = left
@right = right
end
def evaluate(variables = {}) def evaluate(variables = {})
@left.evaluate(variables) != @right.evaluate(variables) @left.evaluate(variables) != @right.evaluate(variables)
end end
...@@ -20,6 +15,10 @@ module Gitlab ...@@ -20,6 +15,10 @@ module Gitlab
def self.build(_value, behind, ahead) def self.build(_value, behind, ahead)
new(behind, ahead) new(behind, ahead)
end end
def self.precedence
10 # See: https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html
end
end end
end end
end end
......
...@@ -8,11 +8,6 @@ module Gitlab ...@@ -8,11 +8,6 @@ module Gitlab
class NotMatches < Lexeme::Operator class NotMatches < Lexeme::Operator
PATTERN = /\!~/.freeze PATTERN = /\!~/.freeze
def initialize(left, right)
@left = left
@right = right
end
def evaluate(variables = {}) def evaluate(variables = {})
text = @left.evaluate(variables) text = @left.evaluate(variables)
regexp = @right.evaluate(variables) regexp = @right.evaluate(variables)
...@@ -23,6 +18,10 @@ module Gitlab ...@@ -23,6 +18,10 @@ module Gitlab
def self.build(_value, behind, ahead) def self.build(_value, behind, ahead)
new(behind, ahead) new(behind, ahead)
end end
def self.precedence
10 # See: https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html
end
end end
end end
end end
......
...@@ -6,9 +6,29 @@ module Gitlab ...@@ -6,9 +6,29 @@ module Gitlab
module Expression module Expression
module Lexeme module Lexeme
class Operator < Lexeme::Base class Operator < Lexeme::Base
# This operator class is design to handle single operators that take two
# arguments. Expression::Parser was originally designed to read infix operators,
# and so the two operands are called "left" and "right" here. If we wish to
# implement an Operator that takes a greater or lesser number of arguments, a
# structural change or additional Operator superclass will likely be needed.
OperatorError = Class.new(Expression::ExpressionError)
def initialize(left, right)
raise OperatorError, 'Invalid left operand' unless left.respond_to? :evaluate
raise OperatorError, 'Invalid right operand' unless right.respond_to? :evaluate
@left = left
@right = right
end
def self.type def self.type
:operator :operator
end end
def self.precedence
raise NotImplementedError
end
end end
end end
end end
......
# frozen_string_literal: true
module Gitlab
module Ci
module Pipeline
module Expression
module Lexeme
class Or < Lexeme::Operator
PATTERN = /\|\|/.freeze
def evaluate(variables = {})
@left.evaluate(variables) || @right.evaluate(variables)
end
def self.build(_value, behind, ahead)
new(behind, ahead)
end
def self.precedence
12 # See: https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html
end
end
end
end
end
end
end
...@@ -8,10 +8,11 @@ module Gitlab ...@@ -8,10 +8,11 @@ module Gitlab
require_dependency 're2' require_dependency 're2'
class Pattern < Lexeme::Value class Pattern < Lexeme::Value
PATTERN = %r{^/.+/[ismU]*$}.freeze PATTERN = %r{^/.+/[ismU]*$}.freeze
NEW_PATTERN = %r{^\/([^\/]|\\/)+[^\\]\/[ismU]*}.freeze
def initialize(regexp) def initialize(regexp)
@value = regexp @value = self.class.eager_matching_with_escape_characters? ? regexp.gsub(/\\\//, '/') : regexp
unless Gitlab::UntrustedRegexp::RubySyntax.valid?(@value) unless Gitlab::UntrustedRegexp::RubySyntax.valid?(@value)
raise Lexer::SyntaxError, 'Invalid regular expression!' raise Lexer::SyntaxError, 'Invalid regular expression!'
...@@ -24,9 +25,17 @@ module Gitlab ...@@ -24,9 +25,17 @@ module Gitlab
raise Expression::RuntimeError, 'Invalid regular expression!' raise Expression::RuntimeError, 'Invalid regular expression!'
end end
def self.pattern
eager_matching_with_escape_characters? ? NEW_PATTERN : PATTERN
end
def self.build(string) def self.build(string)
new(string) new(string)
end end
def self.eager_matching_with_escape_characters?
Feature.enabled?(:ci_variables_complex_expressions, default_enabled: true)
end
end end
end end
end end
......
...@@ -20,6 +20,19 @@ module Gitlab ...@@ -20,6 +20,19 @@ module Gitlab
Expression::Lexeme::NotMatches Expression::Lexeme::NotMatches
].freeze ].freeze
NEW_LEXEMES = [
Expression::Lexeme::Variable,
Expression::Lexeme::String,
Expression::Lexeme::Pattern,
Expression::Lexeme::Null,
Expression::Lexeme::Equals,
Expression::Lexeme::Matches,
Expression::Lexeme::NotEquals,
Expression::Lexeme::NotMatches,
Expression::Lexeme::And,
Expression::Lexeme::Or
].freeze
MAX_TOKENS = 100 MAX_TOKENS = 100
def initialize(statement, max_tokens: MAX_TOKENS) def initialize(statement, max_tokens: MAX_TOKENS)
...@@ -45,7 +58,7 @@ module Gitlab ...@@ -45,7 +58,7 @@ module Gitlab
return tokens if @scanner.eos? return tokens if @scanner.eos?
lexeme = LEXEMES.find do |type| lexeme = available_lexemes.find do |type|
type.scan(@scanner).tap do |token| type.scan(@scanner).tap do |token|
tokens.push(token) if token.present? tokens.push(token) if token.present?
end end
...@@ -58,6 +71,10 @@ module Gitlab ...@@ -58,6 +71,10 @@ module Gitlab
raise Lexer::SyntaxError, 'Too many tokens!' raise Lexer::SyntaxError, 'Too many tokens!'
end end
def available_lexemes
Feature.enabled?(:ci_variables_complex_expressions, default_enabled: true) ? NEW_LEXEMES : LEXEMES
end
end end
end end
end end
......
...@@ -5,17 +5,30 @@ module Gitlab ...@@ -5,17 +5,30 @@ module Gitlab
module Pipeline module Pipeline
module Expression module Expression
class Parser class Parser
ParseError = Class.new(Expression::ExpressionError)
def initialize(tokens) def initialize(tokens)
@tokens = tokens.to_enum @tokens = tokens.to_enum
@nodes = [] @nodes = []
end end
##
# This produces a reverse descent parse tree.
#
# It currently does not support precedence of operators.
#
def tree def tree
if Feature.enabled?(:ci_variables_complex_expressions, default_enabled: true)
rpn_parse_tree
else
reverse_descent_parse_tree
end
end
def self.seed(statement)
new(Expression::Lexer.new(statement).tokens)
end
private
# This produces a reverse descent parse tree.
# It does not support precedence of operators.
def reverse_descent_parse_tree
while token = @tokens.next while token = @tokens.next
case token.type case token.type
when :operator when :operator
...@@ -32,8 +45,51 @@ module Gitlab ...@@ -32,8 +45,51 @@ module Gitlab
@nodes.last || Lexeme::Null.new @nodes.last || Lexeme::Null.new
end end
def self.seed(statement) def rpn_parse_tree
new(Expression::Lexer.new(statement).tokens) results = []
tokens_rpn.each do |token|
case token.type
when :value
results.push(token.build)
when :operator
right_operand = results.pop
left_operand = results.pop
token.build(left_operand, right_operand).tap do |res|
results.push(res)
end
else
raise ParseError, 'Unprocessable token found in parse tree'
end
end
raise ParseError, 'Unreachable nodes in parse tree' if results.count > 1
raise ParseError, 'Empty parse tree' if results.count < 1
results.pop
end
# Parse the expression into Reverse Polish Notation
# (See: Shunting-yard algorithm)
def tokens_rpn
output = []
operators = []
@tokens.each do |token|
case token.type
when :value
output.push(token)
when :operator
if operators.any? && token.lexeme.precedence >= operators.last.lexeme.precedence
output.push(operators.pop)
end
operators.push(token)
end
end
output.concat(operators.reverse)
end end
end end
end end
......
...@@ -7,27 +7,6 @@ module Gitlab ...@@ -7,27 +7,6 @@ module Gitlab
class Statement class Statement
StatementError = Class.new(Expression::ExpressionError) StatementError = Class.new(Expression::ExpressionError)
GRAMMAR = [
# presence matchers
%w[variable],
# positive matchers
%w[variable equals string],
%w[variable equals variable],
%w[variable equals null],
%w[string equals variable],
%w[null equals variable],
%w[variable matches pattern],
# negative matchers
%w[variable notequals string],
%w[variable notequals variable],
%w[variable notequals null],
%w[string notequals variable],
%w[null notequals variable],
%w[variable notmatches pattern]
].freeze
def initialize(statement, variables = {}) def initialize(statement, variables = {})
@lexer = Expression::Lexer.new(statement) @lexer = Expression::Lexer.new(statement)
@variables = variables.with_indifferent_access @variables = variables.with_indifferent_access
...@@ -36,10 +15,6 @@ module Gitlab ...@@ -36,10 +15,6 @@ module Gitlab
def parse_tree def parse_tree
raise StatementError if @lexer.lexemes.empty? raise StatementError if @lexer.lexemes.empty?
unless GRAMMAR.find { |syntax| syntax == @lexer.lexemes }
raise StatementError, 'Unknown pipeline expression!'
end
Expression::Parser.new(@lexer.tokens).tree Expression::Parser.new(@lexer.tokens).tree
end end
...@@ -54,6 +29,7 @@ module Gitlab ...@@ -54,6 +29,7 @@ module Gitlab
end end
def valid? def valid?
evaluate
parse_tree.is_a?(Lexeme::Base) parse_tree.is_a?(Lexeme::Base)
rescue Expression::ExpressionError rescue Expression::ExpressionError
false false
......
require 'fast_spec_helper'
require 'rspec-parameterized'
describe Gitlab::Ci::Pipeline::Expression::Lexeme::And do
let(:left) { double('left', evaluate: nil) }
let(:right) { double('right', evaluate: nil) }
describe '.build' do
it 'creates a new instance of the token' do
expect(described_class.build('&&', left, right)).to be_a(described_class)
end
context 'with non-evaluable operands' do
let(:left) { double('left') }
let(:right) { double('right') }
it 'raises an operator error' do
expect { described_class.build('&&', left, right) }.to raise_error Gitlab::Ci::Pipeline::Expression::Lexeme::Operator::OperatorError
end
end
end
describe '.type' do
it 'is an operator' do
expect(described_class.type).to eq :operator
end
end
describe '.precedence' do
it 'has a precedence' do
expect(described_class.precedence).to be_an Integer
end
end
describe '#evaluate' do
let(:operator) { described_class.new(left, right) }
subject { operator.evaluate }
before do
allow(left).to receive(:evaluate).and_return(left_value)
allow(right).to receive(:evaluate).and_return(right_value)
end
context 'when left and right are truthy' do
where(:left_value, :right_value) do
[true, 1, 'a'].permutation(2).to_a
end
with_them do
it { is_expected.to be_truthy }
it { is_expected.to eq(right_value) }
end
end
context 'when left or right is falsey' do
where(:left_value, :right_value) do
[true, false, nil].permutation(2).to_a
end
with_them do
it { is_expected.to be_falsey }
end
end
context 'when left and right are falsey' do
where(:left_value, :right_value) do
[false, nil].permutation(2).to_a
end
with_them do
it { is_expected.to be_falsey }
it { is_expected.to eq(left_value) }
end
end
end
end
...@@ -5,9 +5,21 @@ describe Gitlab::Ci::Pipeline::Expression::Lexeme::Equals do ...@@ -5,9 +5,21 @@ describe Gitlab::Ci::Pipeline::Expression::Lexeme::Equals do
let(:right) { double('right') } let(:right) { double('right') }
describe '.build' do describe '.build' do
it 'creates a new instance of the token' do context 'with non-evaluable operands' do
expect(described_class.build('==', left, right)) it 'creates a new instance of the token' do
.to be_a(described_class) expect { described_class.build('==', left, right) }
.to raise_error Gitlab::Ci::Pipeline::Expression::Lexeme::Operator::OperatorError
end
end
context 'with evaluable operands' do
it 'creates a new instance of the token' do
allow(left).to receive(:evaluate).and_return('my-string')
allow(right).to receive(:evaluate).and_return('my-string')
expect(described_class.build('==', left, right))
.to be_a(described_class)
end
end end
end end
...@@ -17,23 +29,40 @@ describe Gitlab::Ci::Pipeline::Expression::Lexeme::Equals do ...@@ -17,23 +29,40 @@ describe Gitlab::Ci::Pipeline::Expression::Lexeme::Equals do
end end
end end
describe '.precedence' do
it 'has a precedence' do
expect(described_class.precedence).to be_an Integer
end
end
describe '#evaluate' do describe '#evaluate' do
it 'returns false when left and right are not equal' do let(:operator) { described_class.new(left, right) }
allow(left).to receive(:evaluate).and_return(1)
allow(right).to receive(:evaluate).and_return(2)
operator = described_class.new(left, right) subject { operator.evaluate }
expect(operator.evaluate(VARIABLE: 3)).to eq false before do
allow(left).to receive(:evaluate).and_return(left_value)
allow(right).to receive(:evaluate).and_return(right_value)
end end
it 'returns true when left and right are equal' do context 'when left and right are equal' do
allow(left).to receive(:evaluate).and_return(1) where(:left_value, :right_value) do
allow(right).to receive(:evaluate).and_return(1) [%w(string string)]
end
with_them do
it { is_expected.to eq(true) }
end
end
operator = described_class.new(left, right) context 'when left and right are not equal' do
where(:left_value, :right_value) do
['one string', 'two string'].permutation(2).to_a
end
expect(operator.evaluate(VARIABLE: 3)).to eq true with_them do
it { is_expected.to eq(false) }
end
end end
end end
end end
...@@ -6,9 +6,21 @@ describe Gitlab::Ci::Pipeline::Expression::Lexeme::Matches do ...@@ -6,9 +6,21 @@ describe Gitlab::Ci::Pipeline::Expression::Lexeme::Matches do
let(:right) { double('right') } let(:right) { double('right') }
describe '.build' do describe '.build' do
it 'creates a new instance of the token' do context 'with non-evaluable operands' do
expect(described_class.build('=~', left, right)) it 'creates a new instance of the token' do
.to be_a(described_class) expect { described_class.build('=~', left, right) }
.to raise_error Gitlab::Ci::Pipeline::Expression::Lexeme::Operator::OperatorError
end
end
context 'with evaluable operands' do
it 'creates a new instance of the token' do
allow(left).to receive(:evaluate).and_return('my-string')
allow(right).to receive(:evaluate).and_return('/my-string/')
expect(described_class.build('=~', left, right))
.to be_a(described_class)
end
end end
end end
...@@ -18,63 +30,93 @@ describe Gitlab::Ci::Pipeline::Expression::Lexeme::Matches do ...@@ -18,63 +30,93 @@ describe Gitlab::Ci::Pipeline::Expression::Lexeme::Matches do
end end
end end
describe '.precedence' do
it 'has a precedence' do
expect(described_class.precedence).to be_an Integer
end
end
describe '#evaluate' do describe '#evaluate' do
it 'returns false when left and right do not match' do let(:operator) { described_class.new(left, right) }
allow(left).to receive(:evaluate).and_return('my-string')
allow(right).to receive(:evaluate)
.and_return(Gitlab::UntrustedRegexp.new('something'))
operator = described_class.new(left, right) subject { operator.evaluate }
expect(operator.evaluate).to eq false before do
allow(left).to receive(:evaluate).and_return(left_value)
allow(right).to receive(:evaluate).and_return(right_value)
end end
it 'returns true when left and right match' do context 'when left and right do not match' do
allow(left).to receive(:evaluate).and_return('my-awesome-string') let(:left_value) { 'my-string' }
allow(right).to receive(:evaluate) let(:right_value) { Gitlab::UntrustedRegexp.new('something') }
.and_return(Gitlab::UntrustedRegexp.new('awesome.string$'))
operator = described_class.new(left, right) it { is_expected.to eq(nil) }
expect(operator.evaluate).to eq true
end end
it 'supports matching against a nil value' do context 'when left and right match' do
allow(left).to receive(:evaluate).and_return(nil) let(:left_value) { 'my-awesome-string' }
allow(right).to receive(:evaluate) let(:right_value) { Gitlab::UntrustedRegexp.new('awesome.string$') }
.and_return(Gitlab::UntrustedRegexp.new('pattern'))
it { is_expected.to eq(3) }
end
operator = described_class.new(left, right) context 'when left is nil' do
let(:left_value) { nil }
let(:right_value) { Gitlab::UntrustedRegexp.new('pattern') }
expect(operator.evaluate).to eq false it { is_expected.to eq(nil) }
end end
it 'supports multiline strings' do context 'when left is a multiline string and matches right' do
allow(left).to receive(:evaluate).and_return <<~TEXT let(:left_value) do
My awesome contents <<~TEXT
My awesome contents
My-text-string!
TEXT
end
let(:right_value) { Gitlab::UntrustedRegexp.new('text-string') }
it { is_expected.to eq(24) }
end
My-text-string! context 'when left is a multiline string and does not match right' do
TEXT let(:left_value) do
<<~TEXT
My awesome contents
allow(right).to receive(:evaluate) My-terrible-string!
.and_return(Gitlab::UntrustedRegexp.new('text-string')) TEXT
end
operator = described_class.new(left, right) let(:right_value) { Gitlab::UntrustedRegexp.new('text-string') }
expect(operator.evaluate).to eq true it { is_expected.to eq(nil) }
end end
it 'supports regexp flags' do context 'when a matching pattern uses regex flags' do
allow(left).to receive(:evaluate).and_return <<~TEXT let(:left_value) do
My AWESOME content <<~TEXT
TEXT My AWESOME content
TEXT
end
let(:right_value) { Gitlab::UntrustedRegexp.new('(?i)awesome') }
it { is_expected.to eq(3) }
end
allow(right).to receive(:evaluate) context 'when a non-matching pattern uses regex flags' do
.and_return(Gitlab::UntrustedRegexp.new('(?i)awesome')) let(:left_value) do
<<~TEXT
My AWESOME content
TEXT
end
operator = described_class.new(left, right) let(:right_value) { Gitlab::UntrustedRegexp.new('(?i)terrible') }
expect(operator.evaluate).to eq true it { is_expected.to eq(nil) }
end end
end end
end end
...@@ -5,9 +5,21 @@ describe Gitlab::Ci::Pipeline::Expression::Lexeme::NotEquals do ...@@ -5,9 +5,21 @@ describe Gitlab::Ci::Pipeline::Expression::Lexeme::NotEquals do
let(:right) { double('right') } let(:right) { double('right') }
describe '.build' do describe '.build' do
it 'creates a new instance of the token' do context 'with non-evaluable operands' do
expect(described_class.build('!=', left, right)) it 'creates a new instance of the token' do
.to be_a(described_class) expect { described_class.build('!=', left, right) }
.to raise_error Gitlab::Ci::Pipeline::Expression::Lexeme::Operator::OperatorError
end
end
context 'with evaluable operands' do
it 'creates a new instance of the token' do
allow(left).to receive(:evaluate).and_return('my-string')
allow(right).to receive(:evaluate).and_return('my-string')
expect(described_class.build('!=', left, right))
.to be_a(described_class)
end
end end
end end
...@@ -17,23 +29,45 @@ describe Gitlab::Ci::Pipeline::Expression::Lexeme::NotEquals do ...@@ -17,23 +29,45 @@ describe Gitlab::Ci::Pipeline::Expression::Lexeme::NotEquals do
end end
end end
describe '.precedence' do
it 'has a precedence' do
expect(described_class.precedence).to be_an Integer
end
end
describe '#evaluate' do describe '#evaluate' do
it 'returns true when left and right are not equal' do let(:operator) { described_class.new(left, right) }
allow(left).to receive(:evaluate).and_return(1)
allow(right).to receive(:evaluate).and_return(2)
operator = described_class.new(left, right) subject { operator.evaluate }
expect(operator.evaluate(VARIABLE: 3)).to eq true before do
allow(left).to receive(:evaluate).and_return(left_value)
allow(right).to receive(:evaluate).and_return(right_value)
end end
it 'returns false when left and right are equal' do context 'when left and right are equal' do
allow(left).to receive(:evaluate).and_return(1) using RSpec::Parameterized::TableSyntax
allow(right).to receive(:evaluate).and_return(1)
where(:left_value, :right_value) do
'string' | 'string'
1 | 1
'' | ''
nil | nil
end
with_them do
it { is_expected.to eq(false) }
end
end
operator = described_class.new(left, right) context 'when left and right are not equal' do
where(:left_value, :right_value) do
['one string', 'two string', 1, 2, '', nil, false, true].permutation(2).to_a
end
expect(operator.evaluate(VARIABLE: 3)).to eq false with_them do
it { is_expected.to eq(true) }
end
end end
end end
end end
...@@ -6,9 +6,21 @@ describe Gitlab::Ci::Pipeline::Expression::Lexeme::NotMatches do ...@@ -6,9 +6,21 @@ describe Gitlab::Ci::Pipeline::Expression::Lexeme::NotMatches do
let(:right) { double('right') } let(:right) { double('right') }
describe '.build' do describe '.build' do
it 'creates a new instance of the token' do context 'with non-evaluable operands' do
expect(described_class.build('!~', left, right)) it 'creates a new instance of the token' do
.to be_a(described_class) expect { described_class.build('!~', left, right) }
.to raise_error Gitlab::Ci::Pipeline::Expression::Lexeme::Operator::OperatorError
end
end
context 'with evaluable operands' do
it 'creates a new instance of the token' do
allow(left).to receive(:evaluate).and_return('my-string')
allow(right).to receive(:evaluate).and_return('my-string')
expect(described_class.build('!~', left, right))
.to be_a(described_class)
end
end end
end end
...@@ -18,63 +30,93 @@ describe Gitlab::Ci::Pipeline::Expression::Lexeme::NotMatches do ...@@ -18,63 +30,93 @@ describe Gitlab::Ci::Pipeline::Expression::Lexeme::NotMatches do
end end
end end
describe '.precedence' do
it 'has a precedence' do
expect(described_class.precedence).to be_an Integer
end
end
describe '#evaluate' do describe '#evaluate' do
it 'returns true when left and right do not match' do let(:operator) { described_class.new(left, right) }
allow(left).to receive(:evaluate).and_return('my-string')
allow(right).to receive(:evaluate)
.and_return(Gitlab::UntrustedRegexp.new('something'))
operator = described_class.new(left, right) subject { operator.evaluate }
expect(operator.evaluate).to eq true before do
allow(left).to receive(:evaluate).and_return(left_value)
allow(right).to receive(:evaluate).and_return(right_value)
end end
it 'returns false when left and right match' do context 'when left and right do not match' do
allow(left).to receive(:evaluate).and_return('my-awesome-string') let(:left_value) { 'my-string' }
allow(right).to receive(:evaluate) let(:right_value) { Gitlab::UntrustedRegexp.new('something') }
.and_return(Gitlab::UntrustedRegexp.new('awesome.string$'))
operator = described_class.new(left, right) it { is_expected.to eq(true) }
expect(operator.evaluate).to eq false
end end
it 'supports matching against a nil value' do context 'when left and right match' do
allow(left).to receive(:evaluate).and_return(nil) let(:left_value) { 'my-awesome-string' }
allow(right).to receive(:evaluate) let(:right_value) { Gitlab::UntrustedRegexp.new('awesome.string$') }
.and_return(Gitlab::UntrustedRegexp.new('pattern'))
it { is_expected.to eq(false) }
end
operator = described_class.new(left, right) context 'when left is nil' do
let(:left_value) { nil }
let(:right_value) { Gitlab::UntrustedRegexp.new('pattern') }
expect(operator.evaluate).to eq true it { is_expected.to eq(true) }
end end
it 'supports multiline strings' do context 'when left is a multiline string and matches right' do
allow(left).to receive(:evaluate).and_return <<~TEXT let(:left_value) do
My awesome contents <<~TEXT
My awesome contents
My-text-string!
TEXT
end
let(:right_value) { Gitlab::UntrustedRegexp.new('text-string') }
it { is_expected.to eq(false) }
end
My-text-string! context 'when left is a multiline string and does not match right' do
TEXT let(:left_value) do
<<~TEXT
My awesome contents
allow(right).to receive(:evaluate) My-terrible-string!
.and_return(Gitlab::UntrustedRegexp.new('text-string')) TEXT
end
operator = described_class.new(left, right) let(:right_value) { Gitlab::UntrustedRegexp.new('text-string') }
expect(operator.evaluate).to eq false it { is_expected.to eq(true) }
end end
it 'supports regexp flags' do context 'when a matching pattern uses regex flags' do
allow(left).to receive(:evaluate).and_return <<~TEXT let(:left_value) do
My AWESOME content <<~TEXT
TEXT My AWESOME content
TEXT
end
let(:right_value) { Gitlab::UntrustedRegexp.new('(?i)awesome') }
it { is_expected.to eq(false) }
end
allow(right).to receive(:evaluate) context 'when a non-matching pattern uses regex flags' do
.and_return(Gitlab::UntrustedRegexp.new('(?i)awesome')) let(:left_value) do
<<~TEXT
My AWESOME content
TEXT
end
operator = described_class.new(left, right) let(:right_value) { Gitlab::UntrustedRegexp.new('(?i)terrible') }
expect(operator.evaluate).to eq false it { is_expected.to eq(true) }
end end
end end
end end
require 'fast_spec_helper'
require 'rspec-parameterized'
describe Gitlab::Ci::Pipeline::Expression::Lexeme::Or do
let(:left) { double('left', evaluate: nil) }
let(:right) { double('right', evaluate: nil) }
describe '.build' do
it 'creates a new instance of the token' do
expect(described_class.build('||', left, right)).to be_a(described_class)
end
context 'with non-evaluable operands' do
let(:left) { double('left') }
let(:right) { double('right') }
it 'raises an operator error' do
expect { described_class.build('||', left, right) }.to raise_error Gitlab::Ci::Pipeline::Expression::Lexeme::Operator::OperatorError
end
end
end
describe '.type' do
it 'is an operator' do
expect(described_class.type).to eq :operator
end
end
describe '.precedence' do
it 'has a precedence' do
expect(described_class.precedence).to be_an Integer
end
end
describe '#evaluate' do
let(:operator) { described_class.new(left, right) }
subject { operator.evaluate }
before do
allow(left).to receive(:evaluate).and_return(left_value)
allow(right).to receive(:evaluate).and_return(right_value)
end
context 'when left and right are truthy' do
where(:left_value, :right_value) do
[true, 1, 'a'].permutation(2).to_a
end
with_them do
it { is_expected.to be_truthy }
it { is_expected.to eq(left_value) }
end
end
context 'when left or right is truthy' do
where(:left_value, :right_value) do
[true, false, 'a'].permutation(2).to_a
end
with_them do
it { is_expected.to be_truthy }
end
end
context 'when left and right are falsey' do
where(:left_value, :right_value) do
[false, nil].permutation(2).to_a
end
with_them do
it { is_expected.to be_falsey }
it { is_expected.to eq(right_value) }
end
end
end
end
require 'fast_spec_helper' require 'spec_helper'
describe Gitlab::Ci::Pipeline::Expression::Lexeme::Pattern do describe Gitlab::Ci::Pipeline::Expression::Lexeme::Pattern do
describe '.build' do describe '.build' do
...@@ -30,16 +30,6 @@ describe Gitlab::Ci::Pipeline::Expression::Lexeme::Pattern do ...@@ -30,16 +30,6 @@ describe Gitlab::Ci::Pipeline::Expression::Lexeme::Pattern do
.to eq Gitlab::UntrustedRegexp.new('pattern') .to eq Gitlab::UntrustedRegexp.new('pattern')
end end
it 'is a greedy scanner for regexp boundaries' do
scanner = StringScanner.new('/some .* / pattern/')
token = described_class.scan(scanner)
expect(token).not_to be_nil
expect(token.build.evaluate)
.to eq Gitlab::UntrustedRegexp.new('some .* / pattern')
end
it 'does not allow to use an empty pattern' do it 'does not allow to use an empty pattern' do
scanner = StringScanner.new(%(//)) scanner = StringScanner.new(%(//))
...@@ -68,12 +58,90 @@ describe Gitlab::Ci::Pipeline::Expression::Lexeme::Pattern do ...@@ -68,12 +58,90 @@ describe Gitlab::Ci::Pipeline::Expression::Lexeme::Pattern do
.to eq Gitlab::UntrustedRegexp.new('(?im)pattern') .to eq Gitlab::UntrustedRegexp.new('(?im)pattern')
end end
it 'does not support arbitrary flags' do it 'ignores unsupported flags' do
scanner = StringScanner.new('/pattern/x') scanner = StringScanner.new('/pattern/x')
token = described_class.scan(scanner) token = described_class.scan(scanner)
expect(token).to be_nil expect(token).not_to be_nil
expect(token.build.evaluate)
.to eq Gitlab::UntrustedRegexp.new('pattern')
end
it 'is a eager scanner for regexp boundaries' do
scanner = StringScanner.new('/some .* / pattern/')
token = described_class.scan(scanner)
expect(token).not_to be_nil
expect(token.build.evaluate)
.to eq Gitlab::UntrustedRegexp.new('some .* ')
end
it 'does not match on escaped regexp boundaries' do
scanner = StringScanner.new('/some .* \/ pattern/')
token = described_class.scan(scanner)
expect(token).not_to be_nil
expect(token.build.evaluate)
.to eq Gitlab::UntrustedRegexp.new('some .* / pattern')
end
it 'recognizes \ as an escape character for /' do
scanner = StringScanner.new('/some numeric \/$ pattern/')
token = described_class.scan(scanner)
expect(token).not_to be_nil
expect(token.build.evaluate)
.to eq Gitlab::UntrustedRegexp.new('some numeric /$ pattern')
end
it 'does not recognize \ as an escape character for $' do
scanner = StringScanner.new('/some numeric \$ pattern/')
token = described_class.scan(scanner)
expect(token).not_to be_nil
expect(token.build.evaluate)
.to eq Gitlab::UntrustedRegexp.new('some numeric \$ pattern')
end
context 'with the ci_variables_complex_expressions feature flag disabled' do
before do
stub_feature_flags(ci_variables_complex_expressions: false)
end
it 'is a greedy scanner for regexp boundaries' do
scanner = StringScanner.new('/some .* / pattern/')
token = described_class.scan(scanner)
expect(token).not_to be_nil
expect(token.build.evaluate)
.to eq Gitlab::UntrustedRegexp.new('some .* / pattern')
end
it 'does not recognize the \ escape character for /' do
scanner = StringScanner.new('/some .* \/ pattern/')
token = described_class.scan(scanner)
expect(token).not_to be_nil
expect(token.build.evaluate)
.to eq Gitlab::UntrustedRegexp.new('some .* \/ pattern')
end
it 'does not recognize the \ escape character for $' do
scanner = StringScanner.new('/some numeric \$ pattern/')
token = described_class.scan(scanner)
expect(token).not_to be_nil
expect(token.build.evaluate)
.to eq Gitlab::UntrustedRegexp.new('some numeric \$ pattern')
end
end end
end end
......
...@@ -58,6 +58,56 @@ describe Gitlab::Ci::Pipeline::Expression::Lexer do ...@@ -58,6 +58,56 @@ describe Gitlab::Ci::Pipeline::Expression::Lexer do
expect { lexer.tokens } expect { lexer.tokens }
.to raise_error described_class::SyntaxError .to raise_error described_class::SyntaxError
end end
context 'with complex expressions' do
using RSpec::Parameterized::TableSyntax
subject { described_class.new(expression).tokens.map(&:value) }
where(:expression, :tokens) do
'$PRESENT_VARIABLE =~ /my var/ && $EMPTY_VARIABLE =~ /nope/' | ['$PRESENT_VARIABLE', '=~', '/my var/', '&&', '$EMPTY_VARIABLE', '=~', '/nope/']
'$EMPTY_VARIABLE == "" && $PRESENT_VARIABLE' | ['$EMPTY_VARIABLE', '==', '""', '&&', '$PRESENT_VARIABLE']
'$EMPTY_VARIABLE == "" && $PRESENT_VARIABLE != "nope"' | ['$EMPTY_VARIABLE', '==', '""', '&&', '$PRESENT_VARIABLE', '!=', '"nope"']
'$PRESENT_VARIABLE && $EMPTY_VARIABLE' | ['$PRESENT_VARIABLE', '&&', '$EMPTY_VARIABLE']
'$PRESENT_VARIABLE =~ /my var/ || $EMPTY_VARIABLE =~ /nope/' | ['$PRESENT_VARIABLE', '=~', '/my var/', '||', '$EMPTY_VARIABLE', '=~', '/nope/']
'$EMPTY_VARIABLE == "" || $PRESENT_VARIABLE' | ['$EMPTY_VARIABLE', '==', '""', '||', '$PRESENT_VARIABLE']
'$EMPTY_VARIABLE == "" || $PRESENT_VARIABLE != "nope"' | ['$EMPTY_VARIABLE', '==', '""', '||', '$PRESENT_VARIABLE', '!=', '"nope"']
'$PRESENT_VARIABLE || $EMPTY_VARIABLE' | ['$PRESENT_VARIABLE', '||', '$EMPTY_VARIABLE']
'$PRESENT_VARIABLE && null || $EMPTY_VARIABLE == ""' | ['$PRESENT_VARIABLE', '&&', 'null', '||', '$EMPTY_VARIABLE', '==', '""']
end
with_them do
it { is_expected.to eq(tokens) }
end
end
context 'with the ci_variables_complex_expressions feature flag turned off' do
before do
stub_feature_flags(ci_variables_complex_expressions: false)
end
it 'incorrectly tokenizes conjunctive match statements as one match statement' do
tokens = described_class.new('$PRESENT_VARIABLE =~ /my var/ && $EMPTY_VARIABLE =~ /nope/').tokens
expect(tokens.map(&:value)).to eq(['$PRESENT_VARIABLE', '=~', '/my var/ && $EMPTY_VARIABLE =~ /nope/'])
end
it 'incorrectly tokenizes disjunctive match statements as one statement' do
tokens = described_class.new('$PRESENT_VARIABLE =~ /my var/ || $EMPTY_VARIABLE =~ /nope/').tokens
expect(tokens.map(&:value)).to eq(['$PRESENT_VARIABLE', '=~', '/my var/ || $EMPTY_VARIABLE =~ /nope/'])
end
it 'raises an error about && operators' do
expect { described_class.new('$EMPTY_VARIABLE == "" && $PRESENT_VARIABLE').tokens }
.to raise_error(Gitlab::Ci::Pipeline::Expression::Lexer::SyntaxError).with_message('Unknown lexeme found!')
end
it 'raises an error about || operators' do
expect { described_class.new('$EMPTY_VARIABLE == "" || $PRESENT_VARIABLE').tokens }
.to raise_error(Gitlab::Ci::Pipeline::Expression::Lexer::SyntaxError).with_message('Unknown lexeme found!')
end
end
end end
describe '#lexemes' do describe '#lexemes' do
......
...@@ -2,25 +2,67 @@ require 'fast_spec_helper' ...@@ -2,25 +2,67 @@ require 'fast_spec_helper'
describe Gitlab::Ci::Pipeline::Expression::Parser do describe Gitlab::Ci::Pipeline::Expression::Parser do
describe '#tree' do describe '#tree' do
context 'when using operators' do context 'when using two operators' do
it 'returns a reverse descent parse tree' do
expect(described_class.seed('$VAR1 == "123"').tree)
.to be_a Gitlab::Ci::Pipeline::Expression::Lexeme::Equals
end
end
context 'when using three operators' do
it 'returns a reverse descent parse tree' do it 'returns a reverse descent parse tree' do
expect(described_class.seed('$VAR1 == "123" == $VAR2').tree) expect(described_class.seed('$VAR1 == "123" == $VAR2').tree)
.to be_a Gitlab::Ci::Pipeline::Expression::Lexeme::Equals .to be_a Gitlab::Ci::Pipeline::Expression::Lexeme::Equals
end end
end end
context 'when using a single token' do context 'when using a single variable token' do
it 'returns a single token instance' do it 'returns a single token instance' do
expect(described_class.seed('$VAR').tree) expect(described_class.seed('$VAR').tree)
.to be_a Gitlab::Ci::Pipeline::Expression::Lexeme::Variable .to be_a Gitlab::Ci::Pipeline::Expression::Lexeme::Variable
end end
end end
context 'when using a single string token' do
it 'returns a single token instance' do
expect(described_class.seed('"some value"').tree)
.to be_a Gitlab::Ci::Pipeline::Expression::Lexeme::String
end
end
context 'when expression is empty' do context 'when expression is empty' do
it 'returns a null token' do it 'returns a null token' do
expect(described_class.seed('').tree) expect { described_class.seed('').tree }
.to raise_error Gitlab::Ci::Pipeline::Expression::Parser::ParseError
end
end
context 'when expression is null' do
it 'returns a null token' do
expect(described_class.seed('null').tree)
.to be_a Gitlab::Ci::Pipeline::Expression::Lexeme::Null .to be_a Gitlab::Ci::Pipeline::Expression::Lexeme::Null
end end
end end
context 'when two value tokens have no operator' do
it 'raises a parsing error' do
expect { described_class.seed('$VAR "text"').tree }
.to raise_error Gitlab::Ci::Pipeline::Expression::Parser::ParseError
end
end
context 'when an operator has no left side' do
it 'raises an OperatorError' do
expect { described_class.seed('== "123"').tree }
.to raise_error Gitlab::Ci::Pipeline::Expression::Lexeme::Operator::OperatorError
end
end
context 'when an operator has no right side' do
it 'raises an OperatorError' do
expect { described_class.seed('$VAR ==').tree }
.to raise_error Gitlab::Ci::Pipeline::Expression::Lexeme::Operator::OperatorError
end
end
end end
end end
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment