|
|
|
|
|
|
|
|
|
|
|
This figure shows an example of tolerance issues
|
|
|
|
|
|
|
|
#### Tune the tolerance
|
|
|
|
|
|
|
|
This operation can be done modifying the `.json` associated with the test (tests are located in the tests directory of the svn directory).
|
|
|
|
In the section `comparisons` of the `.json` file, you will find a list of files that have to be compared.
|
|
|
|
For each file, there is a field "method" that is used to define how to compare the reference and the test result files.
|
|
|
|
|
|
|
|
##### "method": "absolute"
|
|
|
|
|
|
|
|
This method compares the absolute difference between two values: `|base-test|`.
|
|
|
|
The field "diff": [float] contains the maximum-allowed difference between two values. If `|base-test| > diff`, the test fails.
|
|
|
|
|
|
|
|
##### "method": "relative"
|
|
|
|
|
|
|
|
This method compares the relative difference between two values: `|(test-base)/base|` or `|(test-base)/test|` if `base=0`. The field "diff": [float] contains the maximum-allowed difference.
|
|
|
|
|
|
|
|
##### "method": "rounding"
|
|
|
|
|
|
|
|
This method is a bit more tricky. It determines approximately the number of significant digits that are taken into account when doing the comparison. This number is defined by the field "tolerance" : [integer]. More exactly, the formula is the following: `diff=base*10^-tolerance` with diff used as for the method absolute, excepted if `tolerance=0`. In this case, no difference is allowed at all.
|
|
|
|
|
|
|
|
##### Choose the right values
|
|
|
|
|
|
|
|
The web page specific to each test file can help you tune the method parameters diff or tolerance:
|
|
|
|
|
|
|
|
* MaxAbsDiff corresponds to the maximum absolute difference detected
|
|
|
|
* MaxRelDiff corresponds to the maximum relative difference detected
|
|
|
|
* MaxTol corresponds to the maximum tolerance detected
|
|
|
|
|
|
|
|
Generally speaking, we advise using the smallest possible value (biggest in the case of tolerance/rounding).
|
|
|
|
Start with the rounding method, which is more generic and fit with values of different ranges.
|
|
|
|
Correct tolerance is bigger than 6 (or 0 in the ideal case). If it's lower, you should use the absolute or the relative methods. If the differences remain significant, question yourself about the relevancy of your results. |
|
|
|
\ No newline at end of file |
