1 Dec 2011 05:19
Re: exact decimal type, was: 12.3*70.95 a little bit weird
Raymond Toy <toy.raymond <at> gmail.com>
2011-12-01 04:19:03 GMT
2011-12-01 04:19:03 GMT
On 11/30/11 9:21 AM, Stavros Macrakis wrote: > How about rat(12.345678)? Of course, ratepsilon should be 2e-15 or so > (which we agreed on in February 2005, but it's still 1e-8!) and you > probably want to turn off ratprint. Just remembered there's one problem with using rat. The number is converted to a floating-point number before rat gets it. So you get FP roundoff. With r notation, it truly is exact and you can have way more than 15 significant digits. Using bfloats is a workaround, which only works if fpprec is bigger than the number of digits you've entered. Ray
RSS Feed