Interesting Difference in Math Calculations in different programs

Discussion in 'Windows 7 Software' started by SteveH, Nov 5, 2012.

  1. SteveH

    SteveH New Member

    Joined:
    Nov 5, 2012
    Messages:
    6
    Likes Received:
    0
    I noticed a difference in the results of a math problem I did in 4 separate programs, and they split 2 to 2 on the result. I multiplied the following 2 numbers in Windows Calculator, a Firefox add-on calculator, Libre Office Calc and MS Excel with interesting resuts :

    82.19777664*33.20088775

    Results
    Windows Calculator = 2,729.039155524212
    Firefox Add-on Calculator = 2729.039155524212
    Libre Office Calc = 2,729.039155524210
    Microsoft Excel 2010 = 2,729.039155524210

    As you can see, the calculators agree on the numbers and the spreadsheet programs agree on the numbers, but as you can also see the 2 different types of programs agreed on different numbers - it appears the spreadsheets rounded the last number down to zero.

    I allowed for 12 decimal places in the spreadsheet programs, I even added decimal places to 14 or 15 and I got the same results - with additional trailing zero's in the additional decimal places in the spreadsheet programs.

    Anyone have any idea how and why these programs are handling the math this way? Thanks for any insight anyone can shed on this riddle
     
  2. cliffordcooley

    cliffordcooley New Member

    Joined:
    Nov 25, 2010
    Messages:
    480
    Likes Received:
    30
    The answer you are looking for deals with numeric precision.

    Numeric precision in Microsoft Excel
    Floating-point arithmetic may give inaccurate results in Excel
    IEEE floating point - (look closely at the number of digits for each level of precision.)
     
  3. SteveH

    SteveH New Member

    Joined:
    Nov 5, 2012
    Messages:
    6
    Likes Received:
    0
    Thanks cliffordcooley, that's an interesting explanation, a great answer to my question:)
     
  4. patcooke

    patcooke Microsoft MVP
    Staff Member Premium Supporter Microsoft MVP

    Joined:
    May 16, 2010
    Messages:
    5,456
    Likes Received:
    268
    An interesting bit of trivia arising out of this question - it is not many years since a pc asked to calculate 1/7 would give the output as 0.142857142857142 . . . . . (recurring ad inf). If the answer were then multplied by 7 it responded with 0.9999999 . . . . which is understandable. Combining the two stages into one however, 1/7*7 gave the same result 0.99999. . .!
     
  5. SteveH

    SteveH New Member

    Joined:
    Nov 5, 2012
    Messages:
    6
    Likes Received:
    0
    That is a very interesting piece of trivia patcooke, I've already learned a couple of interesting facts from this post, thanks for that one :)
     
  6. Joe S

    Joe S Excellent Member

    Joined:
    Jan 12, 2009
    Messages:
    3,785
    Likes Received:
    113
    I'm retired and used to run a CNC machine and in a few instances the rounding caused some real frustration. The machine did it's thing and the prgram they used to create the machine's program did it's job and occasionally there was a puzzling fault that stopped the program. That and the machine's probe used 8 places and displayed values rounded back to 4 but calculated with the 8. My day man and I eventually things figured out and shared the info freely. The programers were pretty clueless!
    Joe
     

Share This Page

Loading...