Bug 53327

Summary: [PATCH] DoubleFormatUtil does not work with huge precision
Product: XMLGraphicsCommons - Now in Jira Reporter: Julien Aymé <julien.ayme>
Component: utilitiesAssignee: XML Graphics Project Mailing List <general>
Status: CLOSED FIXED    
Severity: normal Keywords: PatchAvailable
Priority: P2    
Version: Trunk   
Target Milestone: --   
Hardware: PC   
OS: All   
Attachments: Patch for the test case, which shows the issue
The patched DoubleFormatUtil
patch for both class and test case (DoubleFormatUtil.java and DoubleFormatUtilTest.java)
Patch n.2 for the test case
Patch n.2 for DoubleFormatUtil class

Description Julien Aymé 2012-05-30 08:50:40 UTC
Created attachment 28859 [details]
Patch for the test case, which shows the issue

The new DoubleFormatUtil class fails to format correctly doubles near Double.MIN_VALUE (0 <= value <= 1e-300) with precision between 300 and 340.

I fixed the DoubleFormatUtil class and added a test case which shows the issue.
Comment 1 Julien Aymé 2012-05-30 08:51:18 UTC
Created attachment 28860 [details]
The patched DoubleFormatUtil
Comment 2 Glenn Adams 2012-06-01 19:25:12 UTC
applied at http://svn.apache.org/viewvc?rev=1345315&view=rev

thanks julien, please review and close if satisfied
Comment 3 Julien Aymé 2012-06-02 07:24:50 UTC
Thanks Glenn for applying the patch, everything seems fine after review.
I'm closing the bug.
Comment 4 Glenn Adams 2012-06-02 20:22:18 UTC
(In reply to comment #3)
> Thanks Glenn for applying the patch, everything seems fine after review.
> I'm closing the bug.

I encountered a failure when running junit on XGC today as follows:

Testcase: testAllDoubleRanges took 4.477 sec
        FAILED
value: 5.22534294505995E-4, decimals: 17, precision: 17 expected:<0.00052253429450[6]> but was:<0.00052253429450[599]>
junit.framework.ComparisonFailure: value: 5.22534294505995E-4, decimals: 17, precision: 17 expected:<0.00052253429450[6]> but was:<0.00052253429450[599]>
        at org.apache.xmlgraphics.util.DoubleFormatUtilTest.assertEquals(DoubleFormatUtilTest.java:223)
        at org.apache.xmlgraphics.util.DoubleFormatUtilTest.testAllDoubleRanges(DoubleFormatUtilTest.java:531)

However, on a re-attempt, it did not fail. Perhaps this is because you use random numbers in your test data.
Comment 5 Julien Aymé 2012-06-02 20:58:10 UTC
I'll investigate on this particular value + precision on monday.
Comment 6 Julien Aymé 2012-06-04 10:56:49 UTC
OK, I found the issue: in certain cases, the method "tooCloseTooRound" should return true so that the "formatPrecise" method is used instead of the "formatFast" method. So I tweaked the "tooManyDigitsUsed" and "tooCloseTooRound" methods a bit, so that they perform better.

Patch will follow.
Comment 7 Julien Aymé 2012-06-04 10:58:36 UTC
Created attachment 28880 [details]
patch for both class and test case (DoubleFormatUtil.java and DoubleFormatUtilTest.java)

Enhanced DoubleFormatUtil.tooCloseTooRound + DoubleFormatUtil.tooManyDigitsUsed
Comment 8 Julien Aymé 2012-06-04 11:00:20 UTC
The aim of these two methods ("tooManyDigitsUsed" and "tooCloseTooRound") is to force to use formatPrecise whenever we're not sure that formatFast will produce the right result, but only when formatFast fails, since formatPrecise is much slower (4 times) than formatFast.
Comment 9 Glenn Adams 2012-06-04 19:08:27 UTC
patch applied at http://svn.apache.org/viewvc?rev=1346092&view=rev

thanks julien, please verify and close
Comment 10 Julien Aymé 2012-06-05 08:26:39 UTC
Verified, everything seems OK. Closing bug.
Comment 11 Julien Aymé 2012-06-05 12:40:15 UTC
I found the last (I hope) really rare bugs:
- The first one is due to long <-> double conversion error in DoubleFormatUtil.format private method
- The second is due to a rounding error in double division in DoubleFormatUtil.isRoundedToZero private method when rounding an exact value 5e-N value for certain value of N (when scale = N-1).

Patch for test case and class will follow.
Hopefully I won't find any bug concerning this class anymore.
Comment 12 Julien Aymé 2012-06-05 12:41:02 UTC
Created attachment 28891 [details]
Patch n.2 for the test case
Comment 13 Julien Aymé 2012-06-05 12:41:37 UTC
Created attachment 28892 [details]
Patch n.2 for DoubleFormatUtil class
Comment 14 Glenn Adams 2012-06-05 15:49:30 UTC
apply patch n.2 at http://svn.apache.org/viewvc?rev=1346428&view=rev

thanks julien, please review and close
Comment 15 Julien Aymé 2012-06-11 08:23:31 UTC
At last, everything seems OK !

Thank you again Glenn.