| Key: |
XRP-115
|
| Type: |
Bug
|
| Status: |
Resolved
|
| Resolution: |
Fixed
|
| Priority: |
Critical
|
| Reporter: |
Simone Gianni
|
| Votes: |
0
|
| Watchers: |
1
|
|
If you were logged in you would be able to see more operations.
|
|
xReporter
Created: 24/May/06 08:37 AM
Updated: 10/Sep/07 10:55 AM
|
|
| Component/s: |
None
|
| Affects Version/s: |
1.3
|
| Fix Version/s: |
1.3
|
|
The current code of the DivideFunction does a round half up, thus not returning high precision numbers on divisions. This is due to a subtle difference in java between the BigDecimal function. As a fact, javadocs report that BigDecimal.add :
Returns a BigDecimal whose value is (this + val), and whose scale is max(this.scale(), val.scale()).
This also applies for multiplication and subtraction, but not for division, in this case BigDecimal.divide :
Returns a BigDecimal whose value is (this / val), and whose scale is this.scale(). If rounding must be performed to generate a result with the given scale, the specified rounding mode is applied.
So, 5 (scale 0) / 2 (scale 0) return 3 = (2.5, moved to scale 0 rounding half up).
There are many possible solutions :
a) Division will always return a high precision number, so 5/2=2.5 and 10/3=3.33333333333
--- PRO : we have high precision numbers, as stated in documentation
--- CON : no compatibility with old applications which expect 5/2 = 10/3 = 3
--- CON : ugly number capuld appear (something like 3.333333333333 or 1.5634342972893E-1) where currently nice (but wrong) numbers appear.
a.1) Division will always return a high precision number, but with a (configurable) limit of decimals so 5/2=2.5 and 10/3=3.3333.
--- PRO : we have high precision numbers, as stated in documentation
--- CON : no compatibility with old applications which expect 5/2 = 10/3 = 3
a.2) Division will have configurable rounding mode and number of decimals.
--- PRO : we have high precision numbers if the users wants to
--- PRO : we keep compatibility with older applications, cause defaults will be ROUND_HALF_UP and 0
--- CON : where is the right place to put this configuration? ExpressionContext? In that case it must be configured in code, and not in the expression itself.
b) Division will return the Math.max(scale.a, scale.b) as result scale.
--- PRO : better compatibility with older applications (5/2=3, but 5/a is a = 2.1 will return 2.4...).
--- PRO : configuration in the expression, 5/2.0 = 2.5, 10/3.00 = 3.33, 10/3.000 = 3.333 etc..
--- CON : 10/3 = 3, 10/3.0 = 3.3, 10/3.00 = 3.33 is at least uncommon
c) Add a new operator, for example backslash, to obtain high precision numbers, so 5/2 = 3, 5\2 = 2.5
--- PRO : compatibility with older applications.
--- PRO : per expression decision
They are all quite simple to implement.
|
|
Simone,
just for our information, are you planning to work on this?
In case you missed it, we had a vote on this on the list:
http://lists.cocoondev.org/pipermail/xreporter/2006-May/002682.html
Made the division operator work with a scale of 10. Trailing zeros are removed from the final result.
Added a new integer division function, the notation for this is backslash, e.g. 5 \ 3
This is somewhat different from what was voted on, as that was to let \ be the old behavior, which was to use the precision of the first operand.
SVN rev 691.
|
|
Division rounds half the result, so that 5/2=3