Design and Implementation of the Big Data in Finance and Deep Learning Platform

本研究主旨是希望提供一個智能金融演算法交易平台，以Django CMS作為網頁框架，區分成研發環境與交易環境，完整的功能包含用戶研發、用戶測試以及使用演算法服務。用戶研發與測試上採用IPython的互動式開發介面，利用JupyterHub進行管理與配置，能夠同時提供多個用戶存取平台，使得平台足以負載大規模用戶的使用；而演算法服務經由Celery包裝成任務，以利交付給後台進行分散式運算。搭上近年來深度學習的熱潮，平台額外擴充Tensorflow套件與GPU建置，支援多核及高速演算法運算。 面對存取大量、複雜且結構化的金融資料，本研究的資料庫採用HAWQ做為解決方案，利用其極大量平行化的架構，改善過往存取大數據所造成的系統複雜性與效能瓶頸，並搭配Ambari達到創建、監視及管理Hadoop分散式集群的功用，讓開發者在部署與維運上都將事半功倍。 由於採用新的資料庫HAWQ，傳統的資料表設計將不利反傷，因此本研究會針對程式端存取資料庫裡的金融資料，量身打造適合的資料表設計，並對其做效能評測，以確保資料能有效且迅速地被程式所取用。The purpose of this research is to provide a smartly algorithmic trading platform with financial data. I use Django CMS as a web framework and consisting of Develop environment and Trade environment. The entire functions of the platform include “User Research and Development”,” User Testing” and “Algorithmic Services”. “User Research and Development” and “User Testing” using IPython interactive development interface, with JupyterHub management and configuration, can simultaneously provide multiple user accessing and make the platform enough to support more and more users; “Algorithmic Services” using Celery to package algorithms into tasks can facilitate the delivery to the Server for distributed computing. By means of the growth of Deep Learning in recent years, the platform adds extra Tensorflow and GPU deployment to support multi-core and high-speed algorithm computing. In face of accessing large number of complex and structured financial data, I choose HAWQ as the database in this research. Its extremely massively parallel processing can alleviate the complexity of system and the bottlenecks of efficiency caused by accessing massive number of data. Combing HAWQ with Ambari can achieve the functions of creation, monitoring and management of Hadoop distributed cluster. The developers will do much more easily in deployment and maintenance. The traditional table design may not fit in with the new database HAWQ, so this research will design appropriate table, and evaluate its performance to ensure that data can be accessed effectively and quickly from programs