January 29, 2021 at 1:42 pm
I'm finding that in 2016 the script included at the bottom will return 3.27 for the value 3.275 but I would expect it to be 3.28 because the 5 in the thousandths decimal place would round up. The other two values act as expected where a 4 rounds down and a 6 rounds up. Any version older than 2016 returns 3.28. Does anyone know why this would be?
CREATE TABLE dbo.NumbersTbl (Cost FLOAT)
INSERT INTO dbo.NumbersTbl
VALUES (5.274)
,(3.275)
,(9.276)
SELECT Cost
,convert(NUMERIC(5, 2), Cost)
FROM dbo.NumbersTbl
January 29, 2021 at 4:30 pm
I am guessing here - but it looks to be an issue with using float. Since float is an approximate data type - the value of 3.275 is probably set to 3.27499999999...which is less than 5 and therefore rounded down.
Jeffrey Williams
Problems are opportunities brilliantly disguised as insurmountable obstacles.
How to post questions to get better answers faster
Managing Transaction Logs
January 29, 2021 at 6:21 pm
That part makes sense. Why does 2016 work differently than previous versions?
January 29, 2021 at 6:23 pm
Different hardware - different OS - generating a different float value which is probably 3.275111111111111 or some other variant.
Jeffrey Williams
Problems are opportunities brilliantly disguised as insurmountable obstacles.
How to post questions to get better answers faster
Managing Transaction Logs
January 29, 2021 at 6:35 pm
Understood. I wanted to make sure this wasn't part of some bigger issue or something of that nature. Thanks for the help.
Viewing 5 posts - 1 through 4 (of 4 total)
You must be logged in to reply to this topic. Login to reply