Arithmetic operation failure. Is this a bug, or am I doing something wrong?

Hot Topic

Sql server 2005 June CTP cannot manage databases
SQL 2000 Stored Procedure
varchar (max) - is this correct
Transaction-Log Reader Subsystem errors on: The process could not execute 'sp_replcmds'
still cannot run my job
AS2005 September CTP Memory Error
Appending to a recordset destination
Using Alter View in sproc?
combining 2 select with count and datediff into 1 select. need help.
External program (C#) to call and modify variables of a SSIS package?

SQL Server

High CPU Usage SSIS Package
Newbie: sql 2005 encryption --- or where do I put that key?
Report Builder
CREATE TABLE/VIEW from stored procedure or SELECT...
Restore database from Yukon June CTP to September CTP
What's my login?
Passing Collections to Custom Code
Converting identity column to "not for replication"
Design Ragged Hierarchy in AS 2005
Cannot connect to SQL Server 2005 April CTP from remote computer

Arithmetic operation failure. Is this a bug, or am I doing something wrong?

I used the following code in a function. The @infloat is input parameter

For every other number than 16.31, 17.31, 18.31, 19.31 and 20.31 (at least those numbers I checked) the code runs ok. For those numbers the result is wrong.

What I want to do is return a ten digits number left filled with zeros, with the last two digits consitered as decimals.

What I do is multiply the given float with 100.00, and take the resulting integer

But when @infloat is multiplied with 100.00 and @infloat in (16.31, 17.31, 18.31, 19.31 and 20.31)

the result is (1630,1730,1830,1930,2030) instead of (1631,1731,1831,1931,2031)..

Can anyone try this

declare @ingood int,

@stuffme varchar(10),

@infloat float, @infl float

set @infloat=18.31

set @infloat=isnull(@infloat,0)

set @stuffme='0000000000'

set @infl=@infloat*(100.00)

select @infl as infl

set @ingood=cast(@infl as int)

select @ingood as ingood

set @stuffme=stuff(@stuffme,11-len(@ingood),len(@ingood),@ingood)

select @stuffme




Answer this question

Arithmetic operation failure. Is this a bug, or am I doing something wrong?

  • Be_Ta

    Hi

    I have tried MONEY and NUMERIC datatypes for @infloat and @infl from your example. They both work just fine.

  • someoneontheinternet

    Replace

    set @ingood=cast(@infl as int)

    WITH

    set @ingood=cast(@infl as NUMeERIC (10,2))

  • bunschoten

    Do you need high precision Floating point data is approximate; not all values in the data type range can be precisely represented.

    See here :
    select convert(float, 18.31)
    Result : 18.309999999999999





  • Arithmetic operation failure. Is this a bug, or am I doing something wrong?
Answer this question