Dynamic artificial neural network system proposal

Tez (Doktora) -- İstanbul Teknik Üniversitesi, Fen Bilimleri Enstitüsü, 2014Thesis (Ph.D.) -- İstanbul Technical University, Institute of Science and Technology, 2014Bu çalışmada öncelikle yazında "öğrenme" olarak geçen üç ayrı bilgi edinimi, yapılan tanımlarla birbirinden ayrılmıştır. Bu bilgi edinimlerinden ilki, ilk defa model oluşturmak için çevredeki gözlemlerin kullanılarak modelde belli bir bilgi birikimi sağlamayı belleme veya kavrama (knowledge accumulation) şeklinde tanımlanmış, ikinci tür bilgi birikimi ise çevrede meydana gelen değişikliklerden dolayı yeni kullanılabilir hale gelen gözlemlerin mevcut bir modele dâhil edilmesini sağladığı için uyarlama (adaption) olarak tanımlanmıştır. Üçüncü tür bilgi birikimi ise önceden belli konularda bilgi birikimi elde etmiş veya yetenek kazanmış modellerin, yeni bir konuda bilgi birikimi elde edecek veya mevcut yeteneklerine yeni bir yetenek daha kazanacak şekilde genişletilmesi olarak ifade edilmiş ve geliştirme (enhancement) olarak adlandırılmıştır. Yapay sinir ağları açısından mevcut yazın incelendiğinde kavrama ve uyarlama yapan algoritmaların mevcut olduğu ama geliştirme yapan algoritmaların varlığına veya "geliştirme" kapsamında değerlendirilebilecek bir yaklaşıma rastlanmamaktadır. Yapay sinir ağları açısından "geliştirme" yapan bir algoritmanın hangi işlemler sonucu bunu yapabileceği incelenmiştir. Yapılan incelemede bir algoritmanın yapay sinir ağı açısında geliştirme yapılabilmesi için mutlaka yapısına yeni bir işleve sahip, yeni bir amaca hizmet eden, yeni bir işleve sahip bir bitiş elemanı ekleyebilmesi gerektiği tespit edilmiştir. Bu dönüşüm veya geliştirmelerin zaman içersinde yapay sinir ağının yaptığı işi veya sağladığı faydaları yapısal olarak değiştirdiği için bu algoritmalar dinamik yapay sinir ağı algoritmaları olarak adlandırılmış, bu işlemlerin gerçekleştiği yapay sinir ağı da dinamik yapay sinir ağı olarak adlandırılmıştır. Çalışmanın sınırlarını daha iyi çizebilmek için dinamik yapay sinir ağı kavramının yanında türetilmiş yapay sinir ağı ve bütünleşik yapay sinir ağı kavramları da oluşturulmuştur. Bu çalışmanın konu kapsamı olarak ise yazındaki eksiklik nedeniyle dinamik yapay sinir ağları olarak belirlenmiştir. Dinamik yapay sinir ağı oluşturmak için üç farklı temel algoritma olmak üzere toplam dört yeni algoritma önerilmiştir. Bu çalışmada önerilen algoritmaların performansları ortalama hata karesi, doğru sınıflama yüzdesi ve doğru ikili (binary) fonksiyon tahmini açısından değerlendirilmiştir. Bu değerlendirme sırasında önerilen algoritmalar için farklı çapraz geçerlilik sonuçları ve algoritmaların hangi aday oluşturma yöntemlerini daha çok kullandıkları incelenmiştir. Önerilen algoritmalar, mevcut yazılımlardan SPSS Clementine'nın yapay sinir ağı modeli (dinamik) ve NeuroSolutions v.5.06 yazılımının genetik algoritma ile 25 kromozomun 25 nesil boyunca başlangıç ağırlıklarını, öğrenme oranlarını, momentum oranlarını ve gizli hücre sayısını tarayarak oluşturduğu tek gizli katmana sahip çok katmanlı algılayıcılayıcı modeleri ile karşılaştırılmıştır. Yapılan karşılaştırmalar için üç örnek problem kullanılmıştır. Bu problemler sırasıyla ikili fonksiyon tahminine dayanan ve/veya/özel veya (and/or/xor) mantık problemi, ortalama hata karesini en küçüklenmeye çalışıldığı Wolf'un SunSpot problemi ve son olarak Fisher (1936) tarafından ayırma (discriminant) analizi için kullanılan, türleri doğru sınıflamaya dayanan, Aslanağzı (Iris) Sınıflama problemidir. Bu çalışma kapsamında önerilen dört algoritma da karşılaştırma problemlerinde kayda değer performans göstermiştir. Önerilen dört algoritmadan her problemde ve her performans kriterinde öne çıkan algoritma TA03V02 olurken, en az performans sergileyen algoritma ise TA03V01 olmuştur.In this thesis three different knowledge gain, all referred as "learning" in literature, are differentiated by given definitions. First knowledge gain type called as knowledge accumulation which is the knowledge gain during model building phase using observations from environment. Second type of knowledge gain is referred as adaption, which is the knowledge gain through adding new accrued observations to an existing model due to environment change. Third type of knowledge gain is referred as enhancement, which is the knowledge gain by adding new output properties to an existing working model in such a way that existing model gains a new functionality. When the literature is searched from artificial neural networks point of view, there are algorithms for knowledge accumulation and adaption but there is not any for enhancement. For this reason, the main aim of an algorithm, which does enhancement, is discussed for artificial neural networks. This results in adding new end element(s) in such a way that by adding a new end element the existing artificial neural network gains a new ability or functionality. This transformation or enhancement changes purpose and structure of an existing and working artificial neural network model over time. Thus such algorithms, which enables enhancement, are referred as dynamic artificial neural network algorithms (DANNAs) and such artificial neural networks (ANNs) are referred as dynamic artificial neural networks (DANNs) in this study. To clarify the boundaries of this study, derived artificial neural networks and integrated artificial neural networks are defined as well as dynamic artificial neural networks. Since there are some methods in the literature for derived artificial neural networks and integrated artificial neural networks, this study focuses only on DANNs. To build a dynamic artificial neural network three fundamental algorithms in a total of four dynamic artificial neural network algorithms are proposed. The proposed algorithms in this study are; Fundamental Algorithm 01 Version 01 (FA01V01), Fundamental Algorithm 02 Version 01 (FA02V01), Fundamental Algorithm 03 Version 01 (FA03V01) and Fundamental Algorithm 03 Version 02 (FA03V02). Firstly proposed algorithm FA01V01 only adds new end elements to an existing ANN structure and repeats this prose by adding one end element in each step until the desired number of end elements (output units in some manner) is reached. Second proposed algorithm FA02V01, enables adding processing elements to hidden layer while adding end elements to output layer. FA02V01 repeats this proses by adding one end element in each step until the desired number of end elements (output units in some manner) is reached. FA02V01 differentiates from FA01V01 by evaluating only adding an end element with adding an end element with processing element(s) to hidden layer at the same time. This prevents algorithm to become a decision model that selects to add an end element or processing element in each step. Instead it selects to add best performing structure by evaluating all possible structure alternatives, each consisting an end element. Third proposed fundamental algorithm has two versions (FA03V01 and FA03V02). Fundamental Algorithm 3 (FA03) aims to add an end element to an ANN structured whenever needed. Thus it starts with self-organizing maps (in this study Kohonen's self-organizing map) to identify the patterns in data. Then FA03 uses FA01V01 in FA03V01 to add end elements and FA02V01 in FA03V02 to add end elements and processing elements. When preparing candidate ANN structures, another problem emerges. The first and obvious solution is to prepare a candidate element (end element or processing element) with random weights in its connections. In this study in addition to random weighted elements, eight new methods for end elements and eight new methods for processing elements, in total nine methods for each, are proposed. Some of these proposed methods enable using some of the gained knowledge some of them enables forgetting some of the gained knowledge and some proposed methods enables both using and forgetting some of the gained knowledge. Candidate structure parts, obtained by using methods mentioned above, are collected in a candidate pool and then best performing candidate structure part is selected by the corresponding proposed algorithm according to the predefined performance criteria for the problem. Selecting the best performing candidate structure part is an important decision problem itself. Thus of cross validation is proposed, which is generilized version of -fold cross validation. In a similar manner, proposed of cross validation divides dataset into disjoint subsets as a partition set of dataset, in such a way that each subsets has nearly equal number of elements. In of cross validation sequential set is selected for cross validation. -fold cross validation is a special case of of cross validation in which . Best performing candidate is selected upon performance increase in each cross validation sets . First the number of performance increased sets is counted and the candidates with max number of performance increased cross validation sets are selected. If there is only one candidate, then the best candidate is this sole candidate, but if there is more than one candidate then the candidate which enables the most performance increase in cross validation sets due to performance criteria (mean square error, percentage of correctly classified examples etc.), is selected. The proposed algorithms are evaluated due to performance criteria which are mean square error (MSE), percentage of correctly approximated binary function examples and percentage of correctly classified examples, in this study. Different of cross validation parameters and candidate preparing methods are used and inspected for proposed algorithms to identify which methods are used frequently. Results of the proposed algorithms are compared with ANN (dynamic) model of SPSS Clementine software and 3 layered (one hidden layer) multi-layer perceptron (MLP) model of NeuroSolutions v.5.06 in which starting weights, learning rates, momentum rates and number of processing elements in hidden layer are genetically optimized during 25 generations using 25 chromosomes. For the comparisons three different problems, existing in literature, are used. These problems are; binary function approximation based and/or/xor logic problem, MSE based Wolf's SunSpot problem and Iris Classification Problem, which is used by Fisher (1936) to exemplify discriminant analysis. First problem, and/or/xor logic problem, has two binary inputs and three binary outputs. Due to recitative (20 repeats for each algorithm / model) calculations SPSS Clementine and NeuroSolutions produced models with 100 % correct results. On the other hand, proposed algorithms FA01V01, FA02V01 and FA03V02, produced models with 100 % correct results but not in all twenty repeats. FA03V01 algorithm produced model with maximum 92 % correct results. During this twenty repeats all four algorithms selected best candidates using "correct binary function result percent" as the performance criteria. Second problem used in comparisons is the Wolf's Sunspot dataset, which is the annual record of Sun spots. Caparisons showed that both SPSS Clementine and NeuroSolutions models have higher MSE levels than all four proposed algorithms. In this problem, all four algorithms selected best candidates using "mean square error" as the performance criteria during all twenty repeats. Last problem used in comparisons is the Iris classification problem. Caparisons showed that SPSS Clementine ANN model, NeuroSolutions ANN model, and models of three proposed models (FA01V01, FA02V01 and FA03V02) are reached 100 % correct classification level in test data during twenty repeats. On the other hand proposed FA03V01 algorithm produce ANN models with at most 76.7 % correct classification in test data during twenty repeats. During this twenty repeats all four algorithms selected best candidates using "correct classification percent" as the performance criteria. In this study best of cross validation parameters for each algorithm with respect to each performance criteria is given. A remarkable finding of this study is that 2 of 5 cross validation reduces total number of back-propagation algorithm iterations in all four proposed algorithms. 1 of 10 cross-validation (10-fold cross-validation) is the best parameter for FA01V01 and FA02V01 algorithms with respect to mean square error and correct classification percent. While 10-fold cross-validation is best parameter for FA03V02 with respect to correct classification percent, 1 of 5 cross-validation (5-fold cross-validation) is the best parameter for FA03V02 with respect to mean square error. 2 of 5 cross-validation is the best parameter for FA03V01 algorithm with respect to mean square error, correct classification percent and total number of back-propagation algorithm iterations. A remarkable structure in this study is constructed by FA01V01 for Iris Classification Problem. This 2 PE (processing element) Multi-Layer Perceptron (MLP) structure has 100 % (30/30) correct classifications in test data and 98.33 % (118/120) in training data. This Multi-Layer Perceptron (MLP) structure has 98,67 % (148/150) correct classification in total, which is the at most correct classification level in literature and higher than many machine learning models, with only. 2 PEs (Processing Elements). This MLP structure achieved this performance with 4 Start Elements and a bias element in first layer, 2 Processing Elements and a bias element in hidden layer and 3 End Elements in third layer. All four proposed algorithms showed a remarkable performance in comparison problems. While the most noticeable algorithm is FA03V02, due to overall performance in all three comparison problems, FA03V01 is the least performing algorithm.DoktoraPh.D