A Fast Parallel Stochastic Gradient Method for Matrix Factorization in Shared Memory Systems

在推薦系統上，矩陣分解是一個非常有效的技術。 對於矩陣分解問題，隨機梯度下降法是一個高效的演算法。 然而，這個演算法並不容易被平行。 這篇論文，在共享記憶體系統中，我們開發一個新的平行演算法叫做FPSG。 藉由解決負載不平衡問題及快取失效問題，我們開發的平行演算法比現有的平行演算法更加有效。Matrix factorization is known to be an effective method for recommender systems that are given only the ratings from users to items. Currently, stochastic gradient (SG) method is one of the most popular algorithms for matrix factorization. However, as a sequential approach, SG is difficult to be parallelized for handling web-scale problems. In this thesis, we develop a fast parallel SG method, FPSG, for shared memory systems. By dramatically reducing the cache-miss rate and carefully addressing the load balance of threads, FPSG is more efficient than state-of-the-art parallel algorithms for matrix factorization.TABLE OF CONTENTS 口試委員會審定書................................ i 中文摘要...................................... ii ABSTRACT .................................... iii LISTOFFIGURES................................ vi LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii CHAPTER I.Introduction ............................... 1 II. Existing Parallelized Stochastic Gradient Descent Algorithms andCoordinateDescentMethods.................. 6 2.1 HogWild .............................. 6 2.2 DSGD................................ 8 2.3 CCD++ .............................. 9 III. Problems in Parallel SG Methods for Matrix Factorization . . 13 3.1 LockingProblem.......................... 13 3.2 MemoryDiscontinuity....................... 14 IV.OurApproaches ............................. 16 4.1 Lock-FreeScheduling ....................... 16 4.2 PartialRandomMethod...................... 18 4.3 OverviewofFPSG......................... 21 4.4 ImplementationIssues....................... 22 V.Experiments ............................... 25 5.1 Settings............................... 25 5.2 Comparison of Methods on Training Time versus RMSE . . . . 27 5.2.1 The effectiveness of addressing the locking problem . 27 5.2.2 The effectiveness of having better memory locality . . 28 5.2.3 Comparison with the state-of-the-art methods . . . . 28 5.2.4 Comparison with CCD++ for Non-negative Matrix Factorization....................... 29 5.3 SpeedupofFPSG ......................... 31 VI.Discussion................................. 33 6.1 DataLocalityandtheUpdateOrder............... 33 6.2 NumberofBlocks ......................... 34 VII.ConclusionsandFutureWorks.................... 38 BIBLIOGRAPHY................................. 40 APPENDICES................................... 4