Exploration on Scalability of Database Bulk Insertion with Multi- threading

The advancement of database engine and multi-core processors technologies have enable database insertion to be implemented concurrently via multithreading programming. The objective of this work is to evaluate the performance of using multithreading technique to perform database insertion of large data set with known size to enhance the performance of data access layer (DAL) particularly on the bulk-insertion operation. The performance evaluation includes techniques such as using single database connection, multithreads the insertion process with respective database connections, single threaded bulk insertion and multithreaded bulk insertion. MySQL 5.2 and SQL Server 2008 were used and the experimental results show that the performance of databases do not scale linearly with the number of threads. This work justifies our intention in developing a smart data access layer that autonomously decides the number of threads to spawn in order to improve the performance of th