SDF: Software-Defined Flash for Web-Scale Internet Storage Systems

In the last several years hundreds of thousands of SSDs have been deployed in the data centers of Baidu, China’s largest Internet search company. Currently only 40 % or less of the raw bandwidth of the flash memory in the SSDs is delivered by the storage system to the applications. Moreover, because of space over-provisioning in the SSD to accommodate non-sequential or random writes, and additionally, parity coding across flash channels, typically only 50-70 % of the raw capacity of a commodity SSD can be used for user data. Given the large scale of Baidu’s data center, making the most effective use of its SSDs is of great importance. Specifically, we seek to maximize both bandwidth and usable capacity. To achieve this goal we propose software-defined flash (SDF), a hardware/software co-designed storage system t