|
|
|
Once the testsuite finish, you can access to the results through your internet navigator.
|
|
|
|
To proceed, open the `index.html` file that has been generated in the testsuite directory.
|
|
|
|
|
|
|
|
# Main page
|
|
|
|
|
|
|
|
You will find a table containing the different tests that have been executed, and if their compilation and the test executions have succeeded or failed. You can click on each item to get more information.
|
|
|
|
|
|
|
|
# Compilation
|
|
|
|
|
|
|
|
## Failed compilation
|
|
|
|
|
|
|
|
A failed compilation is shown through a red cross. It indicates that one or several compilations have failed.
|
|
|
|
You can access to the compilation output to get more details and correct the code of Alya to solve the issue.
|
|
|
|
|
|
|
|
# Tests
|
|
|
|
|
|
|
|
## Failed tests
|
|
|
|
|
|
|
|
There are two kinds of failed tests:
|
|
|
|
|
|
|
|
### Hard Failure
|
|
|
|
|
|
|
|
It is shown by a red cross. it means that:
|
|
|
|
|
|
|
|
* execution did not occur correctly,
|
|
|
|
* result files have not been generated,
|
|
|
|
* result files content do not match reference files,
|
|
|
|
* etc.
|
|
|
|
|
|
|
|
In this case, click on the failed tests to get more details about the issue.
|
|
|
|
|
|
|
|
### Tolerance Issue
|
|
|
|
|
|
|
|
This problem is related to the results generated by the execution of Alya. Their value is different from the reference result file values enough to consider that the behavior of Alya is incorrect.
|
|
|
|
You have two options: tune the tolerance if you consider that it is too strict, or modify Alya such that the results are correct.
|
|
|
|
|
|
|
|
#### Tune the tolerance
|
|
|
|
|
|
|
|
This operation can be done modifying the `.json` associated with the test (tests are located in the tests directory of the svn directory).
|
|
|
|
In the section `comparisons` of the `.json` file, you will find a list of files that have to be compared.
|
|
|
|
For each file, there is a field "method" that is used to define how to compare the reference and the test result files.
|
|
|
|
|
|
|
|
##### "method": "absolute"
|
|
|
|
|
|
|
|
This method compares the absolute difference between two values: `|base-test|`.
|
|
|
|
The field "diff": [float] contains the maximum-allowed difference between two values. If `|base-test| > diff`, the test fails.
|
|
|
|
|
|
|
|
##### "method": "relative"
|
|
|
|
|
|
|
|
This method compares the relative difference between two values: `|(test-base)/base|` or `|(test-base)/test|` if `base=0`. The field "diff": [float] contains the maximum-allowed difference.
|
|
|
|
|
|
|
|
##### "method": "rounding"
|
|
|
|
|
|
|
|
This method is a bit more tricky. It determines approximately the number of significant digits that are taken into account when doing the comparison. This number is defined by the field "tolerance" : [integer]. More exactly, the formula is the following: `diff=base*10^-tolerance` with diff used as for the method absolute, excepted if `tolerance=0`. In this case, no difference is allowed at all.
|
|
|
|
|
|
|
|
##### Choose the right values
|
|
|
|
|
|
|
|
The web page specific to each test file can help you tune the method parameters diff or tolerance:
|
|
|
|
|
|
|
|
* MaxAbsDiff corresponds to the maximum absolute difference detected
|
|
|
|
* MaxRelDiff corresponds to the maximum relative difference detected
|
|
|
|
* MaxTol corresponds to the maximum tolerance detected
|
|
|
|
|
|
|
|
Generally speaking, we advise using the smallest possible value (biggest in the case of tolerance/rounding).
|
|
|
|
Start with the rounding method, which is more generic and fit with values of different ranges.
|
|
|
|
Correct tolerance is bigger than 6 (or 0 in the ideal case). If it's lower, you should use the absolute or the relative methods. If the differences remain significant, question yourself about the relevancy of your results.
|
|
|
|
|
|
|
|
|
