Commit 52748867 authored by Dan Davison's avatar Dan Davison

Merge branch 'ml-manage-reusable-resources-as-a-collection' into 'master'

Validate resource reuse after test suite

See merge request gitlab-org/gitlab!78247
parents 6d2c4a85 a88113d6
......@@ -431,11 +431,32 @@ module QA
def validate_reuse_preconditions
raise ResourceReuseError unless reused_name_valid?
end
# Internally we identify an instance of a reusable resource by a unique value of `@reuse_as`, but in GitLab the
# resource has one or more attributes that must also be unique. This method lists those attributes and allows the
# test framework to check that each instance of a reusable resource has values that match the associated values
# in Gitlab.
def unique_identifiers
[:name, :path]
end
end
end
end
```
Reusable resources aren't removed immediately when `remove_via_api!` is called. Instead, they're removed after the test
suite completes. To do so each class must be registered with `QA::Resource::ReusableCollection` in `qa/spec/spec_helper.rb`
as in the example below. Registration allows `QA::Resource::ReusableCollection` to keep track of each instance of each
registered class, and to delete them all in the `config.after(:suite)` hook.
```ruby
config.before(:suite) do |suite|
QA::Resource::ReusableCollection.register_resource_classes do |collection|
QA::Resource::ReusableProject.register(collection)
end
end
```
Consider some examples of how a reusable resource is used:
```ruby
......@@ -488,6 +509,65 @@ end
# match the name specified when the project was first fabricated.
```
### Validating reusable resources
Reusable resources can speed up test suites by avoiding the cost of creating the same resource again and again. However,
that can cause problems if a test makes changes to a resource that prevent it from being reused as expected by later
tests. That can lead to order-dependent test failures that can be difficult to troubleshoot.
For example, the default project created by `QA::Resource::ReusableProject` has `auto_devops_enabled` set to `false`
(inherited from `QA::Resource::Project`). If a test reuses that project and enables Auto DevOps, subsequent tests that reuse
the project will fail if they expect Auto DevOps to be disabled.
We try to avoid that kind of trouble by validating reusable resources after a test suite. If the environment variable
`QA_VALIDATE_RESOURCE_REUSE` is set to `true` the test framework will check each reusable resource to verify that none
of the attributes they were created with have been changed. It does that by creating a new resource using the same
attributes that were used to create the original resource. It then compares the new resource to the original and raises
an error if any attributes don't match.
#### Implementation
When you implement a new type of reusable resource there are two `private` methods you must implement so the resource
can be validated. They are:
- `reference_resource`: creates a new instance of the resource that can be compared with the one that was used during the tests.
- `unique_identifiers`: returns an array of attributes that allow the resource to be identified (e.g., name) and that are therefore
expected to differ when comparing the reference resource with the resource reused in the tests.
The following example shows the implementation of those two methods in `QA::Resource::ReusableProject`.
```ruby
# Creates a new project that can be compared to a reused project, using the attributes of the original.
#
# @return [QA::Resource] a new instance of Resource::ReusableProject that should be a copy of the original resource
def reference_resource
# These are the attributes that the reused resource was created with
attributes = self.class.resources[reuse_as][:attributes]
# Two projects can't have the same path, and since we typically use the same value for the name and path, we assign
# a unique name and path to the reference resource.
name = "reference_resource_#{SecureRandom.hex(8)}_for_#{attributes.delete(:name)}"
Project.fabricate_via_api! do |project|
self.class.resources[reuse_as][:attributes].each do |attribute_name, attribute_value|
project.instance_variable_set("@#{attribute_name}", attribute_value) if attribute_value
end
project.name = name
project.path = name
project.path_with_namespace = "#{project.group.full_path}/#{project.name}"
end
end
# The attributes of the resource that should be the same whenever a test wants to reuse a project.
#
# @return [Array<Symbol>] the attribute names.
def unique_identifiers
# As noted above, path must be unique, and since we typically use the same value for both, we treat name and path
# as unique. These attributes are ignored when we compare the reference and reused resources.
[:name, :path]
end
```
## Where to ask for help?
If you need more information, ask for help on `#quality` channel on Slack
......
......@@ -81,8 +81,6 @@ module QA
Support::FabricationTracker.start_fabrication
result = yield.tap do
fabrication_time = Time.now - start
identifier = resource_identifier(resource)
fabrication_http_method = if resource.api_fabrication_http_method == :get
if include?(Reusable)
"Retrieved for reuse"
......@@ -96,7 +94,7 @@ module QA
Support::FabricationTracker.save_fabrication(:"#{fabrication_method}_fabrication", fabrication_time)
Tools::TestResourceDataProcessor.collect(
resource: resource,
info: identifier,
info: resource.identifier,
fabrication_method: fabrication_method,
fabrication_time: fabrication_time
)
......@@ -104,7 +102,7 @@ module QA
Runtime::Logger.debug do
msg = ["==#{'=' * parents.size}>"]
msg << "#{fabrication_http_method} a #{Rainbow(name).black.bg(:white)}"
msg << identifier
msg << resource.identifier
msg << "as a dependency of #{parents.last}" if parents.any?
msg << "via #{fabrication_method}"
msg << "in #{fabrication_time} seconds"
......@@ -117,26 +115,6 @@ module QA
result
end
# Unique resource identifier
#
# @param [QA::Resource::Base] resource
# @return [String]
def resource_identifier(resource)
if resource.respond_to?(:username) && resource.username
"with username '#{resource.username}'"
elsif resource.respond_to?(:full_path) && resource.full_path
"with full_path '#{resource.full_path}'"
elsif resource.respond_to?(:name) && resource.name
"with name '#{resource.name}'"
elsif resource.respond_to?(:id) && resource.id
"with id '#{resource.id}'"
elsif resource.respond_to?(:iid) && resource.iid
"with iid '#{resource.iid}'"
end
rescue QA::Resource::Base::NoValueError
nil
end
# Define custom attribute
#
# @param [Symbol] name
......@@ -220,6 +198,35 @@ module QA
JSON.pretty_generate(comparable)
end
def diff(other)
return if self == other
diff_values = self.comparable.to_a - other.comparable.to_a
diff_values.to_h
end
def identifier
if respond_to?(:username) && username
"with username '#{username}'"
elsif respond_to?(:full_path) && full_path
"with full_path '#{full_path}'"
elsif respond_to?(:name) && name
"with name '#{name}'"
elsif respond_to?(:id) && id
"with id '#{id}'"
elsif respond_to?(:iid) && iid
"with iid '#{iid}'"
end
rescue QA::Resource::Base::NoValueError
nil
end
def remove_via_api!
super
Runtime::Logger.debug(["Removed a #{self.class.name}", identifier].compact.join(' '))
end
protected
# Custom resource comparison logic using resource attributes from api_resource
......
......@@ -3,12 +3,16 @@
module QA
module Resource
class Group < GroupBase
attributes :require_two_factor_authentication, :description
attributes :require_two_factor_authentication, :description, :path
attribute :full_path do
determine_full_path
end
attribute :name do
@name || path
end
attribute :sandbox do
Sandbox.fabricate_via_api! do |sandbox|
sandbox.api_client = api_client
......@@ -60,7 +64,7 @@ module QA
{
parent_id: sandbox.id,
path: path,
name: path,
name: name || path,
visibility: 'public',
require_two_factor_authentication: @require_two_factor_authentication,
avatar: avatar
......
......@@ -7,6 +7,8 @@ module QA
class GroupBase < Base
include Members
MAX_NAME_LENGTH = 255
attr_accessor :path, :avatar
attributes :id,
......
......@@ -17,6 +17,7 @@ module QA
attributes :id,
:name,
:path,
:add_name_uuid,
:description,
:runners_token,
......
......@@ -4,8 +4,13 @@ module QA
module Resource
#
# This module includes methods that allow resource classes to be reused safely. It should be prepended to a new
# reusable version of an existing resource class. See Resource::Project and ReusableResource::Project for an example
# reusable version of an existing resource class. See Resource::Project and ReusableResource::Project for an example.
# Reusable resource classes must also be registered with a resource collection that will manage cleanup.
#
# @example Register a resource class with a collection
# QA::Resource::ReusableCollection.register_resource_classes do |collection|
# QA::Resource::ReusableProject.register(collection)
# end
module Reusable
attr_accessor :reuse,
:reuse_as
......@@ -16,7 +21,7 @@ module QA
base.extend(ClassMethods)
end
# Gets an existing resource if it exists and the parameters of the new specification of the resource are valid.
# Gets an existing resource if it exists and the specified attributes of the resource are valid.
# Creates a new instance of the resource if it does not exist.
#
# @return [String] The URL of the resource.
......@@ -27,35 +32,128 @@ module QA
rescue Errors::ResourceNotFoundError
super
ensure
self.class.resources[reuse_as] = self
self.class.resources[reuse_as] ||= {
tests: Set.new,
resource: self
}
self.class.resources[reuse_as][:attributes] ||= all_attributes.each_with_object({}) do |attribute_name, attributes|
attributes[attribute_name] = instance_variable_get("@#{attribute_name}")
end
self.class.resources[reuse_as][:tests] << Runtime::Example.location
end
# Including classes must confirm that the resource can be reused as defined. For example, a project can't be
# fabricated with a unique name.
# Overrides remove_via_api! to log a debug message stating that removal will happen after the suite completes.
#
# @return [nil]
def remove_via_api!
QA::Runtime::Logger.debug("#{self.class.name} - deferring removal until after suite")
end
# Object comparison
#
# @param [QA::Resource::Base] other
# @return [Boolean]
def ==(other)
self.class <= other.class && comparable == other.comparable
end
# Confirms that reuse of the resource did not change it in a way that breaks later reuse.
# For example, this should fail if a reusable resource should have a specific name, but the name has been changed.
def validate_reuse
QA::Runtime::Logger.debug(["Validating a #{self.class.name} that was reused as #{reuse_as}", identifier].compact.join(' '))
fresh_resource = reference_resource
diff = reuse_validation_diff(fresh_resource)
if diff.present?
raise ResourceReuseError, <<~ERROR
The reused #{self.class.name} resource does not have the attributes expected.
The following change was found: #{diff}"
The resource's web_url is #{web_url}.
It was used in these tests: #{self.class.resources[reuse_as][:tests].to_a.join(', ')}
ERROR
end
ensure
fresh_resource.remove_via_api!
end
private
# Creates a new resource that can be compared to a reused resource, using the post body of the original.
# Must be implemented by classes that include this module.
def reference_resource
return super if defined?(super)
raise NotImplementedError
end
# Confirms that the resource attributes specified in its fabricate_via_api! block will allow it to be reused.
#
# @return [nil] returns nil unless an error is raised
def validate_reuse_preconditions
return unless self.class.resources.key?(reuse_as)
attributes = unique_identifiers.each_with_object({ proposed: {}, existing: {} }) do |id, attrs|
proposed = public_send(id)
existing = self.class.resources[reuse_as][:resource].public_send(id)
next if proposed == existing
attrs[:proposed][id] = proposed
attrs[:existing][id] = existing
end
unless attributes[:proposed].empty? && attributes[:existing].empty?
raise ResourceReuseError, "Reusable resources must use the same unique identifier(s). " \
"The #{self.class.name} to be reused as :#{reuse_as} has the identifier(s) #{attributes[:proposed]} " \
"but it should have #{attributes[:existing]}"
end
end
# Compares the attributes of the current reused resource with a reference instance.
#
# @return [Hash] any differences between the resources.
def reuse_validation_diff(other)
original, reference = prepare_reuse_validation_diff(other)
return if original == reference
diff_values = original.to_a - reference.to_a
diff_values.to_h
end
# Compares the current reusable resource to a reference instance, ignoring identifying unique attributes that
# had to be changed.
#
# @return [Hash, Hash] the current and reference resource attributes, respectively.
def prepare_reuse_validation_diff(other)
original = self.reload!.comparable
reference = other.reload!.comparable
unique_identifiers.each { |id| reference[id] = original[id] }
[original, reference]
end
# The attributes of the resource that should be the same whenever a test wants to reuse a resource. Must be
# implemented by classes that include this module.
#
# @return [Array<Symbol>] the attribute names.
def unique_identifiers
return super if defined?(super)
raise NotImplementedError
end
module ClassMethods
# Removes all created resources of this type.
#
# @return [Hash<Symbol, QA::Resource>] the resources that were to be removed.
def remove_all_via_api!
resources.each do |reuse_as, resource|
QA::Runtime::Logger.debug("#{self.name} - removing resource reused as :#{reuse_as}")
next QA::Runtime::Logger.debug("#{self.name} reused as :#{reuse_as} has already been removed.") unless resource.exists?
resource.method(:remove_via_api!).super_method.call
end
# Includes the resources created/reused by this class in the specified collection
def register(collection)
collection[self.name] = resources
end
# The resources created by this resource class.
# The resources created/reused by this resource class.
#
# @return [Hash<Symbol, QA::Resource>] the resources created by this resource class.
# @return [Hash<Symbol, Hash>] the resources created/reused by this resource class.
def resources
@resources ||= {}
end
......
# frozen_string_literal: true
require 'singleton'
module QA
module Resource
#
# This singleton class collects all reusable resources used by tests and allows operations to be performed on them
# all. For example, verifying their state after tests have run and might have changed them.
#
class ReusableCollection
include Singleton
attr_accessor :resource_classes
def initialize
@resource_classes = {}
end
# Yields each resource in the collection.
#
# @yieldparam [Symbol] reuse_as the name that identifies the resource instance.
# @yieldparam [QA::Resource] reuse_instance the resource.
def each_resource
resource_classes.each_value do |reuse_instances|
reuse_instances.each do |reuse_as, reuse_instance|
yield reuse_as, reuse_instance[:resource]
end
end
end
class << self
# Removes all created resources that are included in the collection.
def remove_all_via_api!
instance.each_resource do |reuse_as, resource|
next QA::Runtime::Logger.debug("#{resource.class.name} reused as :#{reuse_as} has already been removed.") unless resource.exists?
resource.method(:remove_via_api!).super_method.call
end
end
# Validates the reuse of each resource as defined by the resource class of each resource in the collection.
def validate_resource_reuse
instance.each_resource { |_, resource| resource.validate_reuse }
end
# Yields the collection of resources to allow resource classes to register themselves with the collection.
#
# @yieldparam [Hash] resource_classes the resource classes in the collection.
def register_resource_classes
yield instance.resource_classes
end
end
end
end
end
......@@ -8,46 +8,35 @@ module QA
def initialize
super
@path = "reusable_group"
@name = @path = 'reusable_group'
@description = "QA reusable group"
@reuse_as = :default_group
end
# Confirms that the group can be reused
#
# @return [nil] returns nil unless an error is raised
def validate_reuse_preconditions
unless reused_path_unique?
raise ResourceReuseError,
"Reusable groups must have the same name. The group reused as #{reuse_as} has the path '#{path}' but it should be '#{self.class.resources[reuse_as].path}'"
end
end
private
# Confirms that reuse of the resource did not change it in a way that breaks later reuse. This raises an error if
# the current group path doesn't match the original path.
def validate_reuse
reload!
if api_resource[:path] != @path
raise ResourceReuseError, "The group now has the path '#{api_resource[:path]}' but it should be '#{path}'"
end
end
# Checks if the group is being reused with the same path.
# Creates a new group that can be compared to a reused group, using the attributes of the original. Attributes that
# must be unique (path and name) are replaced with new unique values.
#
# @return [Boolean] true if the group's path is different from another group with the same reuse symbol (reuse_as)
def reused_path_unique?
return true unless self.class.resources.key?(reuse_as)
self.class.resources[reuse_as].path == path
# @return [QA::Resource] a new instance of Resource::ReusableGroup that should be a copy of the original resource
def reference_resource
attributes = self.class.resources[reuse_as][:attributes]
name = "ref#{SecureRandom.hex(8)}_#{attributes.delete(:path)}"[0...MAX_NAME_LENGTH]
Group.fabricate_via_api! do |resource|
self.class.resources[reuse_as][:attributes].each do |attribute_name, attribute_value|
resource.instance_variable_set("@#{attribute_name}", attribute_value) if attribute_value
end
resource.path = name
resource.name = name
end
end
# Overrides QA::Resource::Group#remove_via_api! to log a debug message stating that removal will happen after
# the suite completes rather than now.
# The attributes of the resource that should be the same whenever a test wants to reuse a group.
#
# @return [nil]
def remove_via_api!
QA::Runtime::Logger.debug("#{self.class.name} - deferring removal until after suite")
# @return [Array<Symbol>] the attribute names.
def unique_identifiers
[:name, :path]
end
end
end
......
......@@ -15,36 +15,36 @@ module QA
super
@add_name_uuid = false
@name = "reusable_project"
@name = @path = 'reusable_project'
@reuse_as = :default_project
@initialize_with_readme = true
end
# Confirms that the project can be reused
#
# @return [nil] returns nil unless an error is raised
def validate_reuse_preconditions
unless reused_name_unique?
raise ResourceReuseError,
"Reusable projects must have the same name. The project reused as #{reuse_as} has the name '#{name}' but it should be '#{self.class.resources[reuse_as].name}'"
end
end
private
# Checks if the project is being reused with the same name.
# Creates a new project that can be compared to a reused project, using the attributes of the original. Attributes
# that must be unique (path and name) are replaced with new unique values.
#
# @return [Boolean] true if the project's name is different from another project with the same reuse symbol (reuse_as)
def reused_name_unique?
return true unless self.class.resources.key?(reuse_as)
self.class.resources[reuse_as].name == name
# @return [QA::Resource] a new instance of Resource::ReusableProject that should be a copy of the original resource
def reference_resource
attributes = self.class.resources[reuse_as][:attributes]
name = "reference_resource_#{SecureRandom.hex(8)}_for_#{attributes.delete(:name)}"
Project.fabricate_via_api! do |project|
self.class.resources[reuse_as][:attributes].each do |attribute_name, attribute_value|
project.instance_variable_set("@#{attribute_name}", attribute_value) if attribute_value
end
project.name = name
project.path = name
project.path_with_namespace = "#{project.group.full_path}/#{project.name}"
end
end
# Overrides QA::Resource::Project#remove_via_api! to log a debug message stating that removal will happen after
# the suite completes rather than now.
# The attributes of the resource that should be the same whenever a test wants to reuse a project.
#
# @return [nil]
def remove_via_api!
QA::Runtime::Logger.debug("#{self.class.name} - deferring removal until after suite")
# @return [Array<Symbol>] the attribute names.
def unique_identifiers
[:name, :path]
end
end
end
......
......@@ -454,6 +454,10 @@ module QA
enabled?(ENV['DISABLE_QUARANTINE'], default: false)
end
def validate_resource_reuse?
enabled?(ENV['QA_VALIDATE_RESOURCE_REUSE'], default: false)
end
private
def remote_grid_credentials
......
# frozen_string_literal: true
module QA
module Runtime
module Example
extend self
attr_accessor :current
def location
current.respond_to?(:location) ? current.location : 'unknown'
end
end
end
end
......@@ -3,7 +3,7 @@
RSpec.describe QA::Resource::Base do
include QA::Support::Helpers::StubEnv
let(:resource) { spy('resource', username: 'qa') }
let(:resource) { spy('resource') }
let(:location) { 'http://location' }
let(:log_regex) { %r{==> Built a MyResource with username 'qa' via #{method} in [\d.\-e]+ seconds+} }
......@@ -12,6 +12,36 @@ RSpec.describe QA::Resource::Base do
allow(QA::Tools::TestResourceDataProcessor).to receive(:write_to_file)
end
shared_context 'with simple resource' do
subject do
Class.new(QA::Resource::Base) do
def self.name
'MyResource'
end
attribute :test do
'block'
end
attribute :username do
'qa'
end
attribute :no_block
def fabricate!(*args)
'any'
end
def self.current_url
'http://stub'
end
end
end
let(:resource) { subject.new }
end
shared_context 'with fabrication context' do
subject do
Class.new(described_class) do
......@@ -61,23 +91,29 @@ RSpec.describe QA::Resource::Base do
end
describe '.fabricate_via_api!' do
include_context 'with fabrication context'
context 'when fabricating' do
include_context 'with fabrication context'
it_behaves_like 'fabrication method', :fabricate_via_api!
it_behaves_like 'fabrication method', :fabricate_via_api!
it 'instantiates the resource, calls resource method returns the resource' do
expect(resource).to receive(:fabricate_via_api!).and_return(location)
it 'instantiates the resource, calls resource method returns the resource' do
expect(resource).to receive(:fabricate_via_api!).and_return(location)
result = subject.fabricate_via_api!(resource: resource, parents: [])
result = subject.fabricate_via_api!(resource: resource, parents: [])
expect(result).to eq(resource)
expect(result).to eq(resource)
end
end
context "with debug log level" do
include_context 'with simple resource'
let(:method) { 'api' }
before do
allow(QA::Runtime::Logger).to receive(:debug)
allow(resource).to receive(:api_support?).and_return(true)
allow(resource).to receive(:fabricate_via_api!)
end
it 'logs the resource and build method' do
......@@ -93,27 +129,32 @@ RSpec.describe QA::Resource::Base do
end
describe '.fabricate_via_browser_ui!' do
include_context 'with fabrication context'
context 'when fabricating' do
include_context 'with fabrication context'
it_behaves_like 'fabrication method', :fabricate_via_browser_ui!, :fabricate!
it_behaves_like 'fabrication method', :fabricate_via_browser_ui!, :fabricate!
it 'instantiates the resource and calls resource method' do
subject.fabricate_via_browser_ui!('something', resource: resource, parents: [])
it 'instantiates the resource and calls resource method' do
subject.fabricate_via_browser_ui!('something', resource: resource, parents: [])
expect(resource).to have_received(:fabricate!).with('something')
end
expect(resource).to have_received(:fabricate!).with('something')
end
it 'returns fabrication resource' do
result = subject.fabricate_via_browser_ui!('something', resource: resource, parents: [])
it 'returns fabrication resource' do
result = subject.fabricate_via_browser_ui!('something', resource: resource, parents: [])
expect(result).to eq(resource)
expect(result).to eq(resource)
end
end
context "with debug log level" do
include_context 'with simple resource'
let(:method) { 'browser_ui' }
before do
allow(QA::Runtime::Logger).to receive(:debug)
# allow(resource).to receive(:fabricate!)
end
it 'logs the resource and build method' do
......@@ -128,28 +169,6 @@ RSpec.describe QA::Resource::Base do
end
end
shared_context 'with simple resource' do
subject do
Class.new(QA::Resource::Base) do
attribute :test do
'block'
end
attribute :no_block
def fabricate!
'any'
end
def self.current_url
'http://stub'
end
end
end
let(:resource) { subject.new }
end
describe '.attribute' do
include_context 'with simple resource'
......
# frozen_string_literal: true
RSpec.describe QA::Resource::ReusableCollection do
let(:reusable_resource_class) do
Class.new do
prepend QA::Resource::Reusable
attr_reader :removed
def self.name
'FooReusableResource'
end
def comparable
self.class.name
end
def remove_via_api!
@removed = true
end
def exists?() end
end
end
let(:another_reusable_resource_class) do
Class.new(reusable_resource_class) do
def self.name
'BarReusableResource'
end
end
end
let(:a_resource_instance) { reusable_resource_class.new }
let(:another_resource_instance) { another_reusable_resource_class.new }
it 'is a singleton class' do
expect { described_class.new }.to raise_error(NoMethodError)
end
subject(:collection) do
described_class.instance
end
before do
described_class.register_resource_classes do |c|
reusable_resource_class.register(c)
another_reusable_resource_class.register(c)
end
collection.resource_classes = {
'FooReusableResource' => {
reuse_as_identifier: {
resource: a_resource_instance
}
},
'BarReusableResource' => {
another_reuse_as_identifier: {
resource: another_resource_instance
}
}
}
allow(a_resource_instance).to receive(:validate_reuse)
allow(another_resource_instance).to receive(:validate_reuse)
end
after do
collection.resource_classes = {}
end
describe '#each_resource' do
it 'yields each resource and reuse_as identifier in the collection' do
expect { |blk| collection.each_resource(&blk) }
.to yield_successive_args(
[:reuse_as_identifier, a_resource_instance],
[:another_reuse_as_identifier, another_resource_instance]
)
end
end
describe '.remove_all_via_api!' do
before do
allow(a_resource_instance).to receive(:exists?).and_return(true)
allow(another_resource_instance).to receive(:exists?).and_return(true)
end
it 'removes each instance of each resource class' do
described_class.remove_all_via_api!
expect(a_resource_instance.removed).to be true
expect(another_resource_instance.removed).to be true
end
end
describe '.validate_resource_reuse' do
it 'validates each instance of each resource class' do
expect(a_resource_instance).to receive(:validate_reuse)
expect(another_resource_instance).to receive(:validate_reuse)
described_class.validate_resource_reuse
end
end
describe '.register_resource_classes' do
it 'yields the hash of resource classes in the collection' do
expect { |blk| described_class.register_resource_classes(&blk) }.to yield_with_args(collection.resource_classes)
end
end
end
......@@ -28,8 +28,16 @@ RSpec.configure do |config|
config.add_formatter QA::Support::Formatters::QuarantineFormatter
config.add_formatter QA::Support::Formatters::TestStatsFormatter if QA::Runtime::Env.export_metrics?
config.before(:suite) do |suite|
QA::Resource::ReusableCollection.register_resource_classes do |collection|
QA::Resource::ReusableProject.register(collection)
QA::Resource::ReusableGroup.register(collection)
end
end
config.prepend_before do |example|
QA::Runtime::Logger.debug("\nStarting test: #{example.full_description}\n")
QA::Runtime::Example.current = example
# Reset fabrication counters tracked in resource base
Thread.current[:api_fabrication] = 0
......@@ -66,11 +74,15 @@ RSpec.configure do |config|
end
config.after(:suite) do |suite|
# If any tests failed, leave the resources behind to help troubleshoot
QA::Resource::ReusableProject.remove_all_via_api! unless suite.reporter.failed_examples.present?
# Write all test created resources to JSON file
QA::Tools::TestResourceDataProcessor.write_to_file
# If requested, confirm that resources were used appropriately (e.g., not left with changes that interfere with
# further reuse)
QA::Resource::ReusableCollection.validate_resource_reuse if QA::Runtime::Env.validate_resource_reuse?
# If any tests failed, leave the resources behind to help troubleshoot, otherwise remove them.
QA::Resource::ReusableCollection.remove_all_via_api! unless suite.reporter.failed_examples.present?
end
config.append_after(:suite) do
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment