Subtraction of a decimal less then 1

I am not sure if anyone else had this problem, if so please help...

While writing a simple function that holds the equation:

bVariable = 1 - aVariable

(where: 0 < aVariable < 1)

Every so often a nasty little bug showed its head. I have used every number decleration (keeping both the same). My problem is that the code works, but the value calculated is wrong! When "aVariable" is equal to 0.94, the value calculated for "bVariable" becomes 0.060000000000000000000001

This calculation error only shows up when "aVariable" has the values: 0.94 or 0.97

It is just like the error in win 3.11 where calulations were a "little" off.

Has anyone else come across this Please help, let me know if there is anything else I should try! (I also have this error in vba, java, j++, c++, c#, and pearl)

Answer this question

Subtraction of a decimal less then 1

RickN

MSFT Johan Stenberg wrote:

From http://msdn2.microsoft.com/system.double.aspx:

Remember that a floating-point number can only approximate a decimal number, and that the precision of a floating-point number determines how accurately that number approximates a decimal number. By default, a Double value contains 15 decimal digits of precision, although a maximum of 17 digits is maintained internally.

For a more detailed discussion on floating point arithmetic, see David Goldbergs "What Every Computer Scientist Should Know About Floating-Point Arithmetic":

http://www.validlab.com/goldberg/paper.pdf

He explains the problem quite well when he says that "Squezzing infinitely many real numbers into a finite number requires an approximate representation" :)

Best regards,
Johan Stenberg

Thanks Johan..... I was almost Was off to.....

http://www.thefreecountry.com/compilers/fortran.shtml

It's Interesting.ToMe that This Concept Burnt Trash 80 and Intel a few Years Ago........ Rapter

dmatney
Thank you
PabloL

However, a Decimal has 28 significant digits...are we concluding that when the OP said Decimal, he really meant Double

(And now for something completely different...)
dashbully

How did you declare the variables and assign the values

are they truly decimal, as shakalama, above, declared them
swfranklin

Try expressing 0.06 or 0.04 as a binary number. It is almost as hard as writing pi as a decimal number . Incidentally, if I consistently use Decimal datatypes I don't see this behavior on my machine(s). If I use Double:s, I do get rounding errors...

Please note that you have to use the D suffix for any literals (i.e. 1D - 0.94D)

Best regards,
Johan Stenberg
yavvie

hi,

I have tried this and its just worked fine with/without formation the number

Dim av As Decimal = 0.94

Dim bv As Decimal = Format(1 - av, "0.00")

hope it helps
Microsoft sure makes it hard
Your explanation quotes someone else nicely, but this should not happen this first time I subtract 1 from .96 or .94... so why does it
abcoura

From http://msdn2.microsoft.com/system.double.aspx:

Remember that a floating-point number can only approximate a decimal number, and that the precision of a floating-point number determines how accurately that number approximates a decimal number. By default, a Double value contains 15 decimal digits of precision, although a maximum of 17 digits is maintained internally.

For a more detailed discussion on floating point arithmetic, see David Goldbergs "What Every Computer Scientist Should Know About Floating-Point Arithmetic":

http://www.validlab.com/goldberg/paper.pdf

He explains the problem quite well when he says that "Squeezing infinitely many real numbers into a finite number of bits requires an approximate representation" :)

Best regards,
Johan Stenberg
rctaubert
shakalama wrote:

hi,

I have tried this and its just worked fine with/without formation the number

Dim av As Decimal = 0.94

Dim bv As Decimal = Format(1 - av, "0.00")

hope it helps

Dim av As Double = 0.94

Dim bv As Double = Format(1 - av, "0.000000000000000000000000000000000")

MsgBox(bv)

Oh My God..... Double Is Broke...........
Subtraction of a decimal less then 1

Answer this question