# Interesting Difference in Math Calculations in different programs

Discussion in 'Windows 7 Software' started by SteveH, Nov 5, 2012.

1. ### SteveH New Member

Joined:
Nov 5, 2012
Messages:
6
0
I noticed a difference in the results of a math problem I did in 4 separate programs, and they split 2 to 2 on the result. I multiplied the following 2 numbers in Windows Calculator, a Firefox add-on calculator, Libre Office Calc and MS Excel with interesting resuts :

82.19777664*33.20088775

Results
Windows Calculator = 2,729.039155524212
Firefox Add-on Calculator = 2729.039155524212
Libre Office Calc = 2,729.039155524210
Microsoft Excel 2010 = 2,729.039155524210

As you can see, the calculators agree on the numbers and the spreadsheet programs agree on the numbers, but as you can also see the 2 different types of programs agreed on different numbers - it appears the spreadsheets rounded the last number down to zero.

I allowed for 12 decimal places in the spreadsheet programs, I even added decimal places to 14 or 15 and I got the same results - with additional trailing zero's in the additional decimal places in the spreadsheet programs.

Anyone have any idea how and why these programs are handling the math this way? Thanks for any insight anyone can shed on this riddle

2. ### cliffordcooley New Member

Joined:
Nov 25, 2010
Messages:
480
30
The answer you are looking for deals with numeric precision.

Numeric precision in Microsoft Excel
Floating-point arithmetic may give inaccurate results in Excel
IEEE floating point - (look closely at the number of digits for each level of precision.)

3. ### SteveH New Member

Joined:
Nov 5, 2012
Messages:
6
0
Thanks cliffordcooley, that's an interesting explanation, a great answer to my question

4. ### patcooke Microsoft MVPStaff MemberPremium SupporterMicrosoft MVP

Joined:
May 16, 2010
Messages:
5,533
284
An interesting bit of trivia arising out of this question - it is not many years since a pc asked to calculate 1/7 would give the output as 0.142857142857142 . . . . . (recurring ad inf). If the answer were then multplied by 7 it responded with 0.9999999 . . . . which is understandable. Combining the two stages into one however, 1/7*7 gave the same result 0.99999. . .!

5. ### SteveH New Member

Joined:
Nov 5, 2012
Messages:
6
0
That is a very interesting piece of trivia patcooke, I've already learned a couple of interesting facts from this post, thanks for that one

6. ### Joe S Excellent Member

Joined:
Jan 12, 2009
Messages:
3,792
116
I'm retired and used to run a CNC machine and in a few instances the rounding caused some real frustration. The machine did it's thing and the prgram they used to create the machine's program did it's job and occasionally there was a puzzling fault that stopped the program. That and the machine's probe used 8 places and displayed values rounded back to 4 but calculated with the 8. My day man and I eventually things figured out and shared the info freely. The programers were pretty clueless!
Joe