Performance Optimization of the Samba Read Service on Linux-based Network-Attached Storage Systems

近年來，網路儲存系統（network-attached storage，NAS）廣泛應用於公司或住家內，用來提供檔案、印表機等資源的分享。然而，大多數低階的網路儲存系統處理器為了節省成本，使用了低成本的控制器（controller），這些控制器的運算速度不快，不足夠去處理網路協定與I/O操作，以致於無法有效處理1Gbps乙太（Ethernet）網路甚至10Gbps乙太網路，而產生了效能的瓶頸。了減少處理器的負擔，過去有RDMA（Remote Direct Memory Access）的研究，欲減少處理器花費在資料搬移的時間；也有TCP卸載引擎（TCP offload engine）的研究，透過硬體來分擔處理器處理TCP網路協定的運算。雖然能夠有效減輕處理器的負擔，但缺點是耗費的成本很高、增加系統的複雜度以及軟體必須對應卸載引擎作更改，對於低階的網路儲存系統並不適用。論文對Linux網路儲存系統上Samba伺服器的效能做評估與測試，找出其效能瓶頸，探討如何以較低成本的方式，改善Samba伺服器的檔案讀取效能，以應用在低階的網路儲存系統。我們成功將Samba處理資料的主要功能遷移到Linux作業系統核心（kernel），以減輕對處理器在資料搬移的負擔。由於軟體架構的改變，我們得到5.5% ~ 27.9%的讀取效能增益。In recent years, Network-Attached Storage (NAS) has become a popular choice for storing and sharing in a workgroup or at home. For low-end NAS servers, the tradeoffs between performance and cost is an important issue. Many low-end NAS servers failed to fully utilize the bandwidth of 1Gbps Ethernet network, because the processor embedded in those servers do not have sufficient computation power to process the network protocols and I/O operations. Several solutions were proposed: RDMA (Remote Direct Memory Access) supports zero-copy networking by enabling the network adapter to transfer data directly to or from application memory; TCP offload engine (TOE) shifts TCP/IP processing tasks to the network adapter or storage device. However, even with these solutions, hardware-software co-design is a key issue and software modifications are needed to take advantage from the offload engines. In this thesis, we present a study which uses performance evaluation tools and methodologies to guide software modifications and hardware-software co-design. In this case study, we evaluated the performance of read operations on a Samba server. Then, the performance was improved by porting the application into kernel space after identifying the performance bottleneck by tracing tools. Furthermore, we removed one unnecessary data copy operation for each read request. Our results showed that the performance was improved by 5.5% ~ 27.9% compared to original code. In addition, we compare the performance results with an existing method, the sendfile system call, which removes two major data copy operations while handling read requests in the Samba service.第一章 序論 1.1 研究動機 1.2 論文架構 2二章 背景知識與相關研究 3.1 背景知識 3.1.1 Samba服務簡介 3.1.2 CIFS/SMB協定 3.2 相關研究 5.2.1 RDMA 6.2.2 TCP卸載引擎 7三章 測量架構與量測工具介紹 9.1 測量架構 9.1.1 實驗平台 10.2 量測工具介紹 11.2.1 Iometer負載生成器 11.2.2 mpstat 13.2.3 VTune效能分析器 14.2.4 strace 14.2.5 Valgrind 15.2.6 Pvtrace 16.2.7 DTrace 17.2.8 SystemTap 18.2.9 Wireshark 20四章 探測效能瓶頸 21.1 追蹤網路封包傳遞情況 21.2 量測Samba伺服器之讀取效能 22.3 檢測系統狀態 24.4 追蹤系統呼叫 27.5 系統呼叫之評估 28五章 效能改善機制 31.1 預測效能改善增益 31.2 評估遷移路徑 32.3 移植函式至核心層 36.4 移植過程的問題 37.4.1 資料結構的同步化 37.4.2 檔案系統快取至socket buffer間的資料複製 38六章 sendfile機制 39.1 sendfile運作方式 39.2 追蹤smbd使用sendfile機制的系統呼叫 41.3 smbd使用sendfile的呼叫路徑 41七章 實驗結果與討論 44.1 實驗一：1 outstanding I/O 44.2 實驗二：32 outstanding I/Os 47.3 實驗三：與sendfile版本之比較 48八章 結論與未來展望 51考文獻 5