After applying a curve fit, I use Formula Entry to compare the results but find that the calculated values do not match up with the original values. Why?

This occurs most often with the Polynomial fit and is most likely a precision issue. If your equation only shows 5 significant digits (the default), using these values in Formula Entry will cause your data to be incorrect. You need to include more precision in the parameter values to get accurate results.

If you go back into your curve fit and select Copy Parameters to Calculator from the View pop-up menu, the parameters will be copied over to the appropriate memory registers in full precision. Since they are stored in the memory registers, you would be able to use m0, m1, m2, etc. in your formula instead of having to type the parameter vale.

Scroll to Top