Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Register
  • Sign in
  • alya-testsuite alya-testsuite
  • Project information
    • Project information
    • Activity
    • Members
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Terraform modules
  • Wiki
    • Wiki
  • Activity
Collapse sidebar
  • alyaalya
  • alya-testsuitealya-testsuite
  • Wiki
  • Analyze the results

Analyze the results · Changes

Page history
Update Analyze the results authored Jul 23, 2019 by Damien Dosimont's avatar Damien Dosimont
Hide whitespace changes
Inline Side-by-side
Analyze-the-results.md
View page @ e34145e9
......@@ -27,6 +27,11 @@ _Output of the failed compilation_
## Failed tests
As for the compilation, you can get more insight into the test results by clicking on the corresponding red cross.
![Capture_d_écran_de_2019-07-23_17-39-20](uploads/86fc414339aaee38f8d4672d3977bff9/Capture_d_écran_de_2019-07-23_17-39-20.png)
_Tests page_
There are two kinds of failed tests:
### Hard Failure
......@@ -38,42 +43,17 @@ It is shown by a red cross. it means that:
* result files content do not match reference files,
* etc.
In this case, click on the failed tests to get more details about the issue.
### Tolerance Issue
This problem is related to the results generated by the execution of Alya. Their value is different from the reference result file values enough to consider that the behavior of Alya is incorrect.
You have two options: tune the tolerance if you consider that it is too strict, or modify Alya such that the results are correct.
#### Tune the tolerance
This operation can be done modifying the `.json` associated with the test (tests are located in the tests directory of the svn directory).
In the section `comparisons` of the `.json` file, you will find a list of files that have to be compared.
For each file, there is a field "method" that is used to define how to compare the reference and the test result files.
In this case, click on the failed tests until you get more details about the issue.
##### "method": "absolute"
![Capture_d_écran_de_2019-07-23_17-42-38](uploads/a8d37c5e8d37639165e9ce3f5ad89320/Capture_d_écran_de_2019-07-23_17-42-38.png)
_This page shows the error type. You can investigate by looking to the corresponding alya, alya2pos and mpio2txt outputs._
This method compares the absolute difference between two values: `|base-test|`.
The field "diff": [float] contains the maximum-allowed difference between two values. If `|base-test| > diff`, the test fails.
##### "method": "relative"
This method compares the relative difference between two values: `|(test-base)/base|` or `|(test-base)/test|` if `base=0`. The field "diff": [float] contains the maximum-allowed difference.
##### "method": "rounding"
This method is a bit more tricky. It determines approximately the number of significant digits that are taken into account when doing the comparison. This number is defined by the field "tolerance" : [integer]. More exactly, the formula is the following: `diff=base*10^-tolerance` with diff used as for the method absolute, excepted if `tolerance=0`. In this case, no difference is allowed at all.
##### Choose the right values
### Tolerance Issue
The web page specific to each test file can help you tune the method parameters diff or tolerance:
It is shown by an orange cross.
* MaxAbsDiff corresponds to the maximum absolute difference detected
* MaxRelDiff corresponds to the maximum relative difference detected
* MaxTol corresponds to the maximum tolerance detected
This problem is related to the results generated by the execution of Alya. Their value is different from the reference result file values enough to consider that the behavior of Alya is incorrect.
You have two options: tune the tolerance if you consider that it is too strict, or modify Alya such that the results are correct. See the https://gitlab.bsc.es/alya/alya-testsuite/wikis/Manage-the-tolerances page to know how to manage the tolerances.
Generally speaking, we advise using the smallest possible value (biggest in the case of tolerance/rounding).
Start with the rounding method, which is more generic and fit with values of different ranges.
Correct tolerance is bigger than 6 (or 0 in the ideal case). If it's lower, you should use the absolute or the relative methods. If the differences remain significant, question yourself about the relevancy of your results.
Clone repository
  • Home
  • User
    • Basic information
    • Execute the Testsuite
    • Analyze the results
  • Developer
    • Adding/modifying tests workflow
    • Manage the tolerances