Floating point arithmetic is… at its worst when it is expected to model the interactions of quantities expressed as decimal strings that are expected to be exact.

I’m running into this problem at work, where adding float values together results in slightly incorrect arithmetic. For example, for a given situation, I’m expecting the value 40.41, but the float value is stored as 40.40995. To work around this, I’ve had to round information to two decimal places after finishing sets of math operations. Kludge!

We will need to think about using decimal values instead of floats due to float’s arithmetic issues.

This entry was posted on Thursday, March 15th, 2007 at 10:57 pm and is filed under Programming. You can follow any responses to this entry through the RSS 2.0 feed.
You can leave a response, or trackback from your own site.

One Response to Floating point numbers and inaccurate arithmetic

[…] Posted by geeknotes under Programming I read Chris Vance’s blog entry about floating point numbers and inaccurate arithmetic. It got me thinking about precision and significant digits, and a simple problem illustrates […]

[…] Posted by geeknotes under Programming I read Chris Vance’s blog entry about floating point numbers and inaccurate arithmetic. It got me thinking about precision and significant digits, and a simple problem illustrates […]