call us today: 1.800.876.8376
email us:
This occurs most often with the Polynomial fit.

  • It is most likely a precision issue. If your equation only shows 5 significant digits (the default), using these values in Formula Entry will cause your data to be incorrect. You need to include more precision in the parameter values to get accurate results.
  • If you go back into your curve fit and select Copy Parameters to Calculator from the View pop-up menu, the parameters will be copied over to the appropriate memory registers in full precision. Since they are stored in the memory registers, you would be able to use m0, m1, m2, etc. in your formula instead of having to type the parameter vale.

Mark as helpful.

Posted in: Curve Fitting: Problems