loss of precision in floating point arithmatic

Created by: subhacom

In floating point arithmetic, the result calculated is sometimes lacks in precision. In the following example, the precision loss is quite severe:

moose #7 (closed)> echo { 10.02e-5/(3.1415921.5*1.5e-10) } 282942

whereas the correct result is: 28294.217991670746

Reported by: subhacom

Created by: subhacom

Logged In: YES user_id=1525879 Originator: NO

This is a messy outcome of rounding. Some of the ill-effects can be fixed using the floatformat command:

moose #9 (closed) > floatformat %9f moose #10 (closed) > echo { 10.02e-5/(3.1415921.5*1.5e-10) } 282942.179917

It is really a genesis parser issue, so I am not inclined to dive much deeper into it.

BTW, Subha's answer is off by a decimal point.

Original comment by: upibhalla

Created by: subhacom

• labels: --> Base system
• assigned_to: nobody --> bhalla
• status: open --> closed-wont-fix

Original comment by: upibhalla