Skip to content

loss of precision in floating point arithmatic

Created by: subhacom

In floating point arithmetic, the result calculated is sometimes lacks in precision. In the following example, the precision loss is quite severe:

moose #7 (closed)> echo { 10.02e-5/(3.1415921.5*1.5e-10) } 282942

whereas the correct result is: 28294.217991670746

Reported by: subhacom