Skip to content

erp5
erp5
Commit b559f4c5 authored by Jérome Perrin's avatar Jérome Perrin
Browse files
Options

test_result: Retry flaky tests 

ERP5 functional tests are failing too often, until we improve the tests or the
code so that we don't have these intermitent failures, restart the failed tests
a bit automatically.

This is of course not good, because it's hiding problems, but the current state
of the test suite with many randomly failing test makes that we sometimes miss
when permanent failures are introduced.

See merge request !1206
parents 06a7737f 435dee38
Hide whitespace changes
Inline Side-by-side
Showing with 1031 additions and 77 deletions
bt5/erp5_test_result/PropertySheetTemplateItem/portal_property_sheets/TestSuite/retry_test_pattern_property.xml 0 → 100644
View file @ b559f4c5
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Standard Property" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>categories</string> </key>
<value>
<tuple>
<string>elementary_type/string</string>
</tuple>
</value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string>Test results whose title match this (regular expression) pattern will be retried once if they fail.</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>retry_test_pattern_property</string> </value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Standard Property</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_test_result/SkinTemplateItem/portal_skins/erp5_test_result/TestResultLine_view.xml
View file @ b559f4c5
......@@ -103,6 +103,7 @@
<string>my_failures</string>
<string>my_errors</string>
<string>my_skips</string>
<string>my_test_result_retry_count</string>
<string>my_string_index</string>
<string>my_translated_simulation_state_title</string>
</list>
......
bt5/erp5_test_result/SkinTemplateItem/portal_skins/erp5_test_result/TestResultLine_view/my_test_result_retry_count.xml 0 → 100644
View file @ b559f4c5
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="IntegerField" module="Products.Formulator.StandardFields"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>id</string> </key>
<value> <string>my_test_result_retry_count</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
<item>
<key> <string>integer_out_of_range</string> </key>
<value> <string>The integer you entered was out of range.</string> </value>
</item>
<item>
<key> <string>not_integer</string> </key>
<value> <string>You did not enter an integer.</string> </value>
</item>
<item>
<key> <string>required_not_found</string> </key>
<value> <string>Input is required but no input given.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_maxwidth</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_width</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>end</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>start</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>whitespace_preserve</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_maxwidth</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_width</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>end</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>input_type</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>start</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>whitespace_preserve</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_maxwidth</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_width</string> </key>
<value> <int>20</int> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>end</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>input_type</string> </key>
<value> <string>text</string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>start</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Retries</string> </value>
</item>
<item>
<key> <string>whitespace_preserve</string> </key>
<value> <int>0</int> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_test_result/SkinTemplateItem/portal_skins/erp5_test_result/TestResult_getTestSuiteData.py
View file @ b559f4c5
"""Returns info about a test result, a mapping containing:
test_suite_relative_url: relative url of test suite
retry_test_pattern: the pattern for test result lines which can be restarted
repository_dict: for each test suite repository, keyed by buildout section id:
revision: commit sha
repository_url: git url of the repository
......@@ -53,5 +54,6 @@ if REQUEST:
return {
'test_suite_relative_url': test_suite.getRelativeUrl(),
'retry_test_pattern': test_suite.getRetryTestPattern(),
'repository_dict': repository_dict,
}
bt5/erp5_test_result/SkinTemplateItem/portal_skins/erp5_test_result/TestResult_statTestResultLineList.py
View file @ b559f4c5
......@@ -4,4 +4,6 @@ return [Object(duration=int(context.getProperty('duration', 0)),
all_tests=context.getProperty('all_tests'),
errors=context.getProperty('errors'),
failures=context.getProperty('failures'),
skips=context.getProperty('skips'))]
skips=context.getProperty('skips'),
test_result_retry_count=context.getProperty('test_result_retry_count', 0),
)]
bt5/erp5_test_result/SkinTemplateItem/portal_skins/erp5_test_result/TestResult_view.xml
View file @ b559f4c5
......@@ -107,6 +107,7 @@
<string>my_failures</string>
<string>my_errors</string>
<string>my_skips</string>
<string>my_test_result_retry_count</string>
<string>my_string_index</string>
<string>my_translated_simulation_state_title</string>
</list>
......
bt5/erp5_test_result/SkinTemplateItem/portal_skins/erp5_test_result/TestResult_view/listbox.xml
View file @ b559f4c5
......@@ -371,6 +371,10 @@
<string>skips</string>
<string>Skips</string>
</tuple>
<tuple>
<string>test_result_retry_count</string>
<string>Retries</string>
</tuple>
<tuple>
<string>string_index</string>
<string>Result</string>
......
bt5/erp5_test_result/SkinTemplateItem/portal_skins/erp5_test_result/TestResult_view/my_test_result_retry_count.xml 0 → 100644
View file @ b559f4c5
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="IntegerField" module="Products.Formulator.StandardFields"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>id</string> </key>
<value> <string>my_test_result_retry_count</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
<item>
<key> <string>integer_out_of_range</string> </key>
<value> <string>The integer you entered was out of range.</string> </value>
</item>
<item>
<key> <string>not_integer</string> </key>
<value> <string>You did not enter an integer.</string> </value>
</item>
<item>
<key> <string>required_not_found</string> </key>
<value> <string>Input is required but no input given.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_maxwidth</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_width</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>end</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>start</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>whitespace_preserve</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_maxwidth</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_width</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>end</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>input_type</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>start</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>whitespace_preserve</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_maxwidth</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_width</string> </key>
<value> <int>20</int> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>end</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>input_type</string> </key>
<value> <string>text</string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>start</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Retries</string> </value>
</item>
<item>
<key> <string>whitespace_preserve</string> </key>
<value> <int>0</int> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_test_result/SkinTemplateItem/portal_skins/erp5_test_result/TestSuite_view.xml
View file @ b559f4c5
......@@ -104,6 +104,7 @@
<string>my_additional_bt5_repository_id</string>
<string>my_source_project_title</string>
<string>my_specialise_title</string>
<string>my_retry_test_pattern</string>
</list>
</value>
</item>
......
bt5/erp5_test_result/SkinTemplateItem/portal_skins/erp5_test_result/TestSuite_view/my_retry_test_pattern.xml 0 → 100644
View file @ b559f4c5
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="StringField" module="Products.Formulator.StandardFields"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>id</string> </key>
<value> <string>my_retry_test_pattern</string> </value>
</item>
<item>
<key> <string>message_values</string> </key>
<value>
<dictionary>
<item>
<key> <string>external_validator_failed</string> </key>
<value> <string>The input failed the external validator.</string> </value>
</item>
<item>
<key> <string>required_not_found</string> </key>
<value> <string>Input is required but no input given.</string> </value>
</item>
<item>
<key> <string>too_long</string> </key>
<value> <string>Too much input was given.</string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>overrides</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_maxwidth</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_width</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>input_type</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>max_length</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>truncate</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>unicode</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>whitespace_preserve</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>tales</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_maxwidth</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_width</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>input_type</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>max_length</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>truncate</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>unicode</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>whitespace_preserve</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</value>
</item>
<item>
<key> <string>values</string> </key>
<value>
<dictionary>
<item>
<key> <string>alternate_name</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>css_class</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>default</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_maxwidth</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>display_width</string> </key>
<value> <int>20</int> </value>
</item>
<item>
<key> <string>editable</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>enabled</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>external_validator</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>extra</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>hidden</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>input_type</string> </key>
<value> <string>text</string> </value>
</item>
<item>
<key> <string>max_length</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>required</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Retry Test Pattern</string> </value>
</item>
<item>
<key> <string>truncate</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>unicode</string> </key>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>whitespace_preserve</string> </key>
<value> <int>0</int> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_test_result/TestTemplateItem/portal_components/test.erp5.testTaskDistribution.py
View file @ b559f4c5
......@@ -7,9 +7,9 @@ import responses
import httplib
class TestTaskDistribution(ERP5TypeTestCase):
class TaskDistributionTestCase(ERP5TypeTestCase):
def afterSetUp(self):
self.portal = portal = self.getPortalObject()
portal = self.portal
self.test_node_module = self.portal.getDefaultModule(portal_type = 'Test Node Module')
self.test_suite_module = self.portal.getDefaultModule(portal_type = 'Test Suite Module')
self.test_result_module = self.portal.getDefaultModule(portal_type = 'Test Result Module')
......@@ -83,8 +83,6 @@ class TestTaskDistribution(ERP5TypeTestCase):
original_performance_class._getTestNodeModule = self._original_getTestNodeModule
original_performance_class._getTestSuiteModule = self._original_getTestSuiteModule
def _createTestNode(self, quantity=1, reference_correction=0,
specialise_value=None):
if specialise_value is None:
......@@ -133,7 +131,76 @@ class TestTaskDistribution(ERP5TypeTestCase):
)
test_suite.validate()
test_suite_list.append(test_suite)
return test_suite_list
return test_suite_list
def _callOptimizeAlarm(self):
self.portal.portal_alarms.task_distributor_alarm_optimize.activeSense()
self.tic()
def _callRestartStuckTestResultAlarm(self):
self.portal.portal_alarms.test_result_alarm_restarted_stuck_test_result.activeSense()
self.tic()
def processTest(self, test_title, revision, start_count=2, stop_count=2,
node_title='Node0'):
"""start_count: number of test line to start
stop_count: number of test line to stop
"""
status_dict = {}
test_result_path, revision = self._createTestResult(revision=revision,
test_list=['testFoo', 'testBar'], test_title=test_title, node_title=node_title)
if start_count:
line_url, test = self.distributor.startUnitTest(test_result_path,
node_title=node_title)
if start_count == 2:
next_line_url, next_test = self.distributor.startUnitTest(test_result_path,
node_title=node_title)
self.assertEqual(set(['testFoo', 'testBar']), set([test, next_test]))
if stop_count:
self.distributor.stopUnitTest(line_url, status_dict, node_title=node_title)
if stop_count == 2:
self.tool.stopUnitTest(next_line_url, status_dict, node_title=node_title)
test_result = self.portal.restrictedTraverse(test_result_path)
self.assertEqual(test_result.getSimulationState(), "started")
self.tic()
if stop_count == 2:
self.assertEquals(test_result.getSimulationState(), "stopped")
else:
self.assertEquals(test_result.getSimulationState(), "started")
def _cleanupTestResult(self):
self.tic()
cleanup_state_list = ['started', 'stopped']
test_list = self.test_result_module.searchFolder(title='"TEST FOO" OR "test suite %" OR "Default Test Suite"',
simulation_state=cleanup_state_list)
for test_result in test_list:
if test_result.getSimulationState() in cleanup_state_list:
test_result.cancel()
test_result.setTitle('previous test')
self.tic()
def _createTestResult(self, revision="r0=a,r1=a", node_title='Node0',
test_list=None, tic=1, allow_restart=False,
test_title=None, distributor=None):
if test_title is None:
test_title = self.default_test_title
if distributor is None:
distributor = self.distributor
result = distributor.createTestResult(
"", revision, test_list or [], allow_restart,
test_title=test_title, node_title=node_title)
# we commit, since usually we have a remote call only doing this
(self.tic if tic else self.commit)()
return result
def checkTestResultLine(self, test_result, expected):
line_list = test_result.objectValues(portal_type="Test Result Line")
found_list = [(x.getTitle(), x.getSimulationState()) for x in line_list]
found_list.sort(key=lambda x: x[0])
self.assertEqual(expected, found_list)
class TestTaskDistribution(TaskDistributionTestCase):
def test_01_createTestNode(self):
test_node = self._createTestNode()[0]
......@@ -198,14 +265,6 @@ class TestTaskDistribution(ERP5TypeTestCase):
self.portal.portal_workflow.doActionFor(test_suite_clone, 'validate_action')
self.assertEqual('validated', test_suite_clone.getValidationState())
def _callOptimizeAlarm(self):
self.portal.portal_alarms.task_distributor_alarm_optimize.activeSense()
self.tic()
def _callRestartStuckTestResultAlarm(self):
self.portal.portal_alarms.test_result_alarm_restarted_stuck_test_result.activeSense()
self.tic()
def test_subscribeNode_ReturnValue(self):
config = self.distributor.subscribeNode('COMP-X', 'QPK')
config = json.loads(config)
......@@ -262,33 +321,6 @@ class TestTaskDistribution(ERP5TypeTestCase):
[('COMP32-Node1',set([u'B1'])), ('COMP32-Node2',set([u'B0']))]],
"%r" % ([config1, config2],))
def processTest(self, test_title, revision, start_count=2, stop_count=2,
node_title='Node0'):
"""start_count: number of test line to start
stop_count: number of test line to stop
"""
status_dict = {}
test_result_path, revision = self._createTestResult(revision=revision,
test_list=['testFoo', 'testBar'], test_title=test_title, node_title=node_title)
if start_count:
line_url, test = self.distributor.startUnitTest(test_result_path,
node_title=node_title)
if start_count == 2:
next_line_url, next_test = self.distributor.startUnitTest(test_result_path,
node_title=node_title)
self.assertEqual(set(['testFoo', 'testBar']), set([test, next_test]))
if stop_count:
self.distributor.stopUnitTest(line_url, status_dict, node_title=node_title)
if stop_count == 2:
self.tool.stopUnitTest(next_line_url, status_dict, node_title=node_title)
test_result = self.portal.restrictedTraverse(test_result_path)
self.assertEqual(test_result.getSimulationState(), "started")
self.tic()
if stop_count == 2:
self.assertEquals(test_result.getSimulationState(), "stopped")
else:
self.assertEquals(test_result.getSimulationState(), "started")
def test_04b_startTestSuiteOrder(self):
"""
When we have many test suites associated to one test nodes, the method
......@@ -498,31 +530,6 @@ class TestTaskDistribution(ERP5TypeTestCase):
finally:
self.unpinDateTime()
def _cleanupTestResult(self):
self.tic()
cleanup_state_list = ['started', 'stopped']
test_list = self.test_result_module.searchFolder(title='"TEST FOO" OR "test suite %" OR "Default Test Suite"',
simulation_state=cleanup_state_list)
for test_result in test_list:
if test_result.getSimulationState() in cleanup_state_list:
test_result.cancel()
test_result.setTitle('previous test')
self.tic()
def _createTestResult(self, revision="r0=a,r1=a", node_title='Node0',
test_list=None, tic=1, allow_restart=False,
test_title=None, distributor=None):
if test_title is None:
test_title = self.default_test_title
if distributor is None:
distributor = self.distributor
result = distributor.createTestResult(
"", revision, test_list or [], allow_restart,
test_title=test_title, node_title=node_title)
# we commit, since usually we have a remote call only doing this
(self.tic if tic else self.commit)()
return result
def test_05_createTestResult(self):
"""
We will check the method createTestResult of distributor
......@@ -675,12 +682,6 @@ class TestTaskDistribution(ERP5TypeTestCase):
['testFailing', 'testSlow', 'testFast'],
[self.tool.startUnitTest(test_result_path)[1] for _ in range(3)])
def checkTestResultLine(self, test_result, expected):
line_list = test_result.objectValues(portal_type="Test Result Line")
found_list = [(x.getTitle(), x.getSimulationState()) for x in line_list]
found_list.sort(key=lambda x: x[0])
self.assertEqual(expected, found_list)
def test_06b_restartStuckTest(self):
"""
Check if a test result line is not stuck in 'started', if so, redraft
......@@ -1417,6 +1418,90 @@ class TestTaskDistribution(ERP5TypeTestCase):
self.assertEqual(None, test_suite.getAlarmDate())
class TestRetryFailedTest(TaskDistributionTestCase):
"""Test how failed tests can be automatically retried.
"""
def afterSetUp(self):
super(TestRetryFailedTest, self).afterSetUp()
self.test_suite, = self.test_suite_module.objectValues()
self._createTestNode()
self.tic()
def test_failed_test_not_retried_by_default(self):
test_result_path, _ = self._createTestResult(test_list=['testFoo', ])
test_result = self.portal.unrestrictedTraverse(test_result_path)
line_url, _ = self.tool.startUnitTest(test_result_path)
test_result_line = self.portal.restrictedTraverse(line_url)
status_dict = {
'test_count': 100,
'error_count': 2,
'failure_count': 3,
}
self.tool.stopUnitTest(line_url, status_dict)
self.tic()
self.assertEqual(test_result_line.getStringIndex(), 'FAILED')
self.assertEqual(test_result_line.getSimulationState(), 'stopped')
self.assertEqual(test_result.getStringIndex(), 'FAIL')
self.assertEqual(test_result.getSimulationState(), 'stopped')
def test_failed_retried_once_then_fail(self):
self.test_suite.setRetryTestPattern('testF.*')
test_result_path, _ = self._createTestResult(test_list=['testFoo', ])
test_result = self.portal.unrestrictedTraverse(test_result_path)
line_url, _ = self.tool.startUnitTest(test_result_path)
test_result_line = self.portal.restrictedTraverse(line_url)
status_dict = {
'test_count': 100,
'error_count': 2,
'failure_count': 3,
}
self.tool.stopUnitTest(line_url, status_dict)
self.tic()
# test failed, but it will be retried
self.assertEqual(test_result_line.getStringIndex(), 'RETRYING')
self.assertEqual(test_result_line.getSimulationState(), 'draft')
# if it fails again ...
self.tool.stopUnitTest(line_url, status_dict)
self.tic()
# ... the test result will be fail.
self.assertEqual(test_result_line.getStringIndex(), 'FAILED')
self.assertEqual(test_result_line.getSimulationState(), 'stopped')
self.assertEqual(test_result.getStringIndex(), 'FAIL')
self.assertEqual(test_result.getSimulationState(), 'stopped')
self.assertEqual(test_result.getProperty('errors'), 2)
self.assertEqual(test_result.getProperty('failures'), 3)
self.assertEqual(test_result.getProperty('test_result_retry_count'), 1)
def test_failed_retried_once_then_pass(self):
self.test_suite.setRetryTestPattern('testF.*')
test_result_path, _ = self._createTestResult(test_list=['testFoo', ])
test_result = self.portal.unrestrictedTraverse(test_result_path)
line_url, _ = self.tool.startUnitTest(test_result_path)
test_result_line = self.portal.restrictedTraverse(line_url)
status_dict = {
'test_count': 100,
'error_count': 2,
'failure_count': 3,
}
self.tool.stopUnitTest(line_url, status_dict)
self.tic()
# test failed, but it will be retried
self.assertEqual(test_result_line.getStringIndex(), 'RETRYING')
self.assertEqual(test_result_line.getSimulationState(), 'draft')
# if it succeed next time ...
status_dict['error_count'] = 0
status_dict['failure_count'] = 0
self.tool.stopUnitTest(line_url, status_dict)
self.tic()
# ... the test result will be successful.
self.assertEqual(test_result_line.getStringIndex(), 'PASSED')
self.assertEqual(test_result_line.getSimulationState(), 'stopped')
self.assertEqual(test_result.getStringIndex(), 'PASS')
self.assertEqual(test_result.getSimulationState(), 'stopped')
self.assertEqual(test_result.getProperty('test_result_retry_count'), 1)
class TestGitlabRESTConnectorInterface(ERP5TypeTestCase):
"""Tests for Gitlab commits annotations.
"""
......@@ -1578,6 +1663,7 @@ class TestGitlabRESTConnectorInterface(ERP5TypeTestCase):
"test_result_line_pattern": None,
},
},
"retry_test_pattern": None,
"test_suite_relative_url": self.test_suite.getRelativeUrl()
},
self.test_result.TestResult_getTestSuiteData())
......
bt5/erp5_test_result/WorkflowTemplateItem/portal_workflow/test_result_workflow/scripts/TestResult_complete.py
View file @ b559f4c5
import re
test_result = sci['object']
kw = sci['kwargs']
test_result.setStopDate(kw.get('date') or DateTime())
......@@ -8,13 +9,38 @@ def unexpected(test_result):
# passed if there's no unexpected failures.
return test_result.getSourceProjectTitle() != "NEO R&D"
def shouldRetry(test_result_line):
# type: (erp5.portal_type.TestResultLine,) -> bool
"""Should the test result line be retried ?
We retry test result line once for tests matching pattern defined on test suite.
Unless if there's already another failed test result line, in that case we don't retry.
"""
if test_result_line.getProperty('test_result_retry_count') or 0:
return False
test_result = test_result_line.getParentValue()
for other_test_result_line in test_result.contentValues(portal_type='Test Result Line'):
if test_result_line != other_test_result_line and other_test_result_line.getStringIndex() in ('UNKNOWN', 'FAILED'):
return False
test_suite_data = test_result.TestResult_getTestSuiteData()
if not test_suite_data:
return False
if not test_suite_data['retry_test_pattern']:
return False
return re.search(test_suite_data['retry_test_pattern'], test_result_line.getTitle() or '')
if test_result.getPortalType() == 'Test Result':
has_unknown_result = False
edit_kw = dict(duration=0,
all_tests=0,
errors=0,
failures=0,
skips=0)
skips=0,
test_result_retry_count=0)
for line in test_result.objectValues(portal_type='Test Result Line'):
for prop in edit_kw:
try:
......@@ -47,12 +73,12 @@ elif test_result.getPortalType() == 'Test Result Line':
duration = kw.get('duration')
if duration is None:
duration = (test_result.getStopDate() - test_result.getStartDate()) * (24*60*60)
cmdline = kw.get('command', getattr(test_result, 'cmdline', ''))
cmdline = kw.get('command', '')
if same_type(cmdline, []):
cmdline = ' '.join(map(repr, cmdline))
stdout = kw.get('stdout', getattr(test_result, 'stdout', ''))
stderr = kw.get('stderr', getattr(test_result, 'stderr', ''))
html_test_result = kw.get('html_test_result', getattr(test_result, 'html_test_result', ''))
stdout = kw.get('stdout', '')
stderr = kw.get('stderr', '')
html_test_result = kw.get('html_test_result', '')
test_result.edit(cmdline=cmdline,
stdout=stdout,
stderr=stderr,
......@@ -63,5 +89,11 @@ elif test_result.getPortalType() == 'Test Result Line':
failures=failures,
skips=skips,
html_test_result=html_test_result)
if status == 'FAILED' and shouldRetry(test_result):
test_result.edit(
test_result_retry_count=1 + (test_result.getProperty('test_result_retry_count') or 0),
string_index='RETRYING',
)
test_result.redraft(comment="Retried after a first failure")
else:
raise NotImplementedError("unknown type : %r" % test_result.getPortalType())
bt5/erp5_test_result/WorkflowTemplateItem/portal_workflow/test_result_workflow/states/stopped.xml
View file @ b559f4c5
......@@ -32,6 +32,7 @@
<string>cancel_action</string>
<string>fail</string>
<string>publish_stopped</string>
<string>redraft</string>
</tuple>
</value>
</item>
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7