以Soft-RoCE在LAN中實現iSCSI over RDMA的效能分析研究;Performance Comparison of iSCSI over RDMA by Soft-RoCE in LAN

[[abstract]]iSCSI融合了SAN的高效能性及NAS的便利性，可透過原有乙太網路當傳輸媒介，是一個低成本且亦易於橫向擴充維護的網路磁碟機協定，非常合適雲端運算的巨大容量及高速擴充高效能的虛擬化架構。但TCP協議極其複雜、編碼量巨大並且充滿了各種特例且很難卸載offload。倘若能將傳統iSCSI實施在TCP上的架構改變至將iSCSI實施在高效低延遲的區網協定上，例如：以Software的方式在原有的乙太網路上來實現iSCSI over RDMA，則不但能解決以往iSCSI效能不佳的問題，也不必因業務需求而要花費大量成本去置換或升級擴充原有區網的設備，而其可橫向擴充及維護成本也遠較一般資訊中心的儲域光纖網路來得有彈性及成本低廉，故以軟體實作的方式來了解其改進效能是一個頗值得探討的方向。為要求得到精確的量測結果，實驗以實際建立測試環境以確認方案可行，且撰寫修改系統核心及驅動程式以深入觀測時間。之後並利用各種效能評量工具實際與NFS及傳統的iSCSI做微觀及巨觀比較其效能，以確認iSCSI over RDMA by Soft-RoCE是否有優於傳統的Sockets的kernelSpace堆疊的NFS/iSCSI應用。實驗證明了iSCSI顯然較NFS有利於小型檔案存取，但卻有可能有部份資料仍並未實際寫入硬碟中，這是iSCSI主要不同於NFS對於client端檔案meta的處理方式。而在使用Verbs API的Soft-RoCE在運作上，實際的數據顯示：在傳輸量變大時，明顯會增加佔用系統CPU資源的消耗，但在Soft-RoCE在執行RDMA的iSER協定上確實改善了在LAN上7%~15%以上的網路磁碟機的延遲的IOPS效能。 結論是在一般的GbE的環境實際測試中，iSER的效能確實有較佳於iSCSI與NFS，更適合已有乙太網路架構以及電子郵件伺服器甚至於具有ERP系統資料庫的一般中小企業內部的檔案分享與應用。關鍵字: LAN、GbE、iSCSI、NFS、RDMA、iSER、Soft-RoCE iSCSI is a storage network technology which combines the performance of SAN and the high scalability of NAS. It is not only compatible with existing Ethernet, but low-cost, scale-out and easy to maintain. It is a good candidate to fulfill the virtualization needs of cloud computing in high capacity、high performance. However, TCP protocol is too complicated, huge and hard to offload. What if we can practice iSCSI on a high performance low latency network protocol rather than TCP, for example: iSCSI over RDMA via Soft-RoCE, then we can solve the poor iSCSI performance issues and benefit from the lower cost to upgrade the existing network. Furthermore, it is far cheaper solution than the traditional data center with fabric. Hence, it is worth to research the software approach work or not. In order to get precise measurements, this experiment actually sets up the test environment to confirm that the approach is feasible and modified system kernel and drivers to deeply observe the performance. Then, we use several assessment tools to actually compare the performance of NFS,iSCSI and iSER in micro、macro. The purpose is to verify the Soft-RoCE is better than the traditional Sockets or not. The experiments demonstrated that iSCSI is clearly better than NFS to access small files, despite some of the data may still not actually written to the hard disk that is iSCSI different from the NFS in file meta approach. Regarding the verbs API in Soft-RoCE, the actual data shows: as the transmission is getting bigger, the system CPU consumption is growing significantly, but the iSER does improve the IOPS of the network storage in 7%~15%. The conclusion is that iSER has better performance than iSCSI and NFS, it is suitable to email or database servers and files sharing in the ERP system of the small and middle model business organization.Keywords: LAN, GbE, iSCSI, NFS, RDMA, iSER, Soft-RoC