Removal of external influences from on-line vis-NIR spectra for predicting soil organic carbon using machine learning

External factors including moisture content negatively affect the prediction accuracy of soil organic carbon (SOC) using on-line visible and near-infrared (vis-NIR) spectroscopy. This study compared the performances of four algorithms to remove the moisture content effect [direct standardization (DS), piecewise direct standardization (PDS), external parameter orthogonalization (EPO), and orthogonal signal correction (OSC)] against non corrected (NC) spectral models developed with partial least squares regression (PLSR), support vector machine (SVM), random forest (RF), and M5Rules regression. An on-line soil sensing platform coupled with a visNIR spectrophotometer (305-1700 nm) was used to scan twelve agricultural fields in Belgium and France. A total of 372 soil samples collected during the on-line measurement were divided into a calibration (260) and a prediction (112) dataset, using the Kennard-Stone algorithm. The latter set together with identical laboratory measured 112 dry soil spectra formed a transfer dataset to develop EPO, DS and PDS correction matrices. Results showed that models after EPO, PDS and OSC corrections resulted in improved accuracy [coefficient of determination (R-2) = 0.60-0.82, root mean square error (RMSE) = 16.1-5.7 g kg(-1))], compared to the NC models (R-2 = 0.58-0.73, RMSE = 16.5-6.8 g kg(-1)), whereas the DS (R-2 =-0.10 to 0.26, RMSE = 26.8-21.9 g kg 1) provided deteriorated prediction accuracy. The EPO and OSC models provided better prediction accuracy than that of the PDS corrected models. The OSC-M5Rules (R-2 = 0.82, RMSE = 5.7 g kg(-1)) obtained the highest accuracy followed by EPO-M5Rules (R-2 = 0.74, RMSE = 6.7 g kg(-1)) and NC-M5Rules (R-2 = 0.73, RMSE = 6.8 g kg(-1)), which outperformed all PLSR, RF and SVM models. Therefore, on-line vis-NIR spectra should be corrected with the OSC algorithm before calibrating a machine learning model for accurate prediction of SOC.External factors including moisture content negatively affect the prediction accuracy of soil organic carbon (SOC) using on-line visible and near-infrared (vis-NIR) spectroscopy. This study compared the performances of four algorithms to remove the moisture content effect [direct standardization (DS), piecewise direct standardization (PDS), external parameter orthogonalization (EPO), and orthogonal signal correction (OSC)] against non corrected (NC) spectral models developed with partial least squares regression (PLSR), support vector machine (SVM), random forest (RF), and M5Rules regression. An on-line soil sensing platform coupled with a visNIR spectrophotometer (305-1700 nm) was used to scan twelve agricultural fields in Belgium and France. A total of 372 soil samples collected during the on-line measurement were divided into a calibration (260) and a prediction (112) dataset, using the Kennard-Stone algorithm. The latter set together with identical laboratory measured 112 dry soil spectra formed a transfer dataset to develop EPO, DS and PDS correction matrices. Results showed that models after EPO, PDS and OSC corrections resulted in improved accuracy [coefficient of determination (R-2) = 0.60-0.82, root mean square error (RMSE) = 16.1-5.7 g kg(-1))], compared to the NC models (R-2 = 0.58-0.73, RMSE = 16.5-6.8 g kg(-1)), whereas the DS (R-2 =-0.10 to 0.26, RMSE = 26.8-21.9 g kg 1) provided deteriorated prediction accuracy. The EPO and OSC models provided better prediction accuracy than that of the PDS corrected models. The OSC-M5Rules (R-2 = 0.82, RMSE = 5.7 g kg(-1)) obtained the highest accuracy followed by EPO-M5Rules (R-2 = 0.74, RMSE = 6.7 g kg(-1)) and NC-M5Rules (R-2 = 0.73, RMSE = 6.8 g kg(-1)), which outperformed all PLSR, RF and SVM models. Therefore, on-line vis-NIR spectra should be corrected with the OSC algorithm before calibrating a machine learning model for accurate prediction of SOC.A