| ... | ... | @@ -6,20 +6,20 @@ Generally, it means that the behavior of Alya is wrong and that you should inves |
|
|
|
|
|
|
|
This operation can be done modifying the `.json` associated with the test (see https://gitlab.bsc.es/alya/alya-testsuite/wikis/Contribution-workflow to know how to proceed).
|
|
|
|
In the section `comparisons` of the `.json` file, you will find a list of files are compared.
|
|
|
|
For each file, there is a field "method" that is used to define how to compare the reference and the test result files.
|
|
|
|
For each file, there is a field "method" that is used to define how to compare the reference and the test result files, and a field "tolerance" that corresponds to the value associated with this method.
|
|
|
|
|
|
|
|
##### "method": "absolute"
|
|
|
|
|
|
|
|
This method compares the absolute difference between two values: `|base-test|`.
|
|
|
|
The field "diff": [float] contains the maximum-allowed difference between two values. If `|base-test| > diff`, the test fails.
|
|
|
|
The field "tolerance": [float] contains the maximum-allowed difference between two values. If `|base-test| > diff`, the test fails.
|
|
|
|
|
|
|
|
##### "method": "relative"
|
|
|
|
|
|
|
|
This method compares the relative difference between two values: `|(test-base)/base|` or `|(test-base)/test|` if `base=0`. The field "diff": [float] contains the maximum-allowed difference.
|
|
|
|
This method compares the relative difference between two values: `|(test-base)/base|` or `|(test-base)/test|` if `base=0`. The field "tolerance": [float] contains the maximum-allowed difference.
|
|
|
|
|
|
|
|
##### "method": "rounding"
|
|
|
|
##### "method": "power"
|
|
|
|
|
|
|
|
This method is a bit more tricky. It determines approximately the number of significant digits that are taken into account when doing the comparison. This number is defined by the field "tolerance" : [integer]. More exactly, the formula is the following: `diff=base*10^-tolerance` with diff used as for the method absolute, excepted if `tolerance=0`. In this case, no difference is allowed at all.
|
|
|
|
This method is a bit more tricky. It determines approximately the number of significant digits that are taken into account when doing the comparison. This number is defined by the field "tolerance" : [integer]. More exactly, the formula is the following: `diff=base*10^-tolerance` and diff used as the method `absolute` tolerance, except if `tolerance=0`. In this case, no difference is allowed at all.
|
|
|
|
|
|
|
|
##### Choose the right values
|
|
|
|
|
| ... | ... | @@ -29,6 +29,6 @@ The web page specific to each test file can help you tune the method parameters |
|
|
|
* MaxRelDiff corresponds to the maximum relative difference detected
|
|
|
|
* MaxTol corresponds to the maximum tolerance detected
|
|
|
|
|
|
|
|
Generally speaking, we advise using the smallest possible value (biggest in the case of tolerance/rounding).
|
|
|
|
Start with the rounding method, which is more generic and fit with values of different ranges.
|
|
|
|
Generally speaking, we advise using the smallest possible value (biggest in the case of tolerance/power).
|
|
|
|
Start with the power method, which is more generic and fit with values of different ranges.
|
|
|
|
Correct tolerance is bigger than 6 (or 0 in the ideal case). If it's lower, you should use the absolute or the relative methods. If the differences remain significant, question yourself about the relevancy of your results. |
|
|
\ No newline at end of file |
