Logistic Regression Methods with Truncated Newton Method

Considering two-class classification, this paper aims to perform further study on the success of Truncated Newton method in Truncated Regularized Kernel Logistic Regression (TR-KLR) and Iterative Re-weighted Least Square (TR-IRLS) on solving the numerical problem of KLR and RLR. The study was conducted by developing the Newton version of TR-KLR and TR-IRLS algorithm respectively. They are general classifiers which are termed respectively as proposed Newton TR-KLR (NTR-KLR) and proposed NTR Regularized Logistic Regression (NTR-LR). Instead of using IRLS procedure as used by TR-KLR and TR-IRLS, the proposed algorithms implement Newton-Raphson method as the outer algorithm of Truncated Newton for KLR and RLR respectively. Since, for KLR and RLR, IRLS is equivalent to Newton-Raphson method, both proposed algorithms can be expected to perform as well as TR-KLR and TR-IRLS. Moreover, both proposed algorithms are mathematically simpler, because they do not need to restate the Newton-Raphson method as the IRLS procedure before such as in TR-KLR and TR-IRLS. Hence, they simply can be applied as further explanation to the effectiveness of Truncated Newton method in TR-KLR and TR-IRLS respectively. Numerical experiment with Image Segmentation data set has demonstrated that proposed NTR-KLR performs effectively when exist the singularity and the training time problem in using Newton-Raphson method for KLR (KLR-NR). While proposed NTR-LR has performed better training time than RLR with Newton-Raphson (RLR-NR) method on Letter Image data set. Moreover, both proposed algorithms have showed consistency with the convergence theory and have promising results, i.e. accurate and stable classification, on image data sets respectively