Connectionist-based intelligent information systems for image analysis and knowledge engineering: applications in horticulture

New Zealand’s main export earnings come from the primary production area including agriculture, horticulture, and viticulture. One of the major contributors in this area of horticulture is the production of quality export grade fruit; specifically apples. In order to maintain a competitive advantage, the systems and methods used to grow the fruit are constantly being refined and are increasingly based on data collected and analysed by both the orchardist who grows the produce and also researchers who refine the methods used to determine high levels of fruit quality. To support the task of data analysis and the resulting decision-making process requires efficient and reliable tools. This thesis attempts to address these issues by applying the techniques of Connectionist-Based Intelligent Information Systems (CBIIS) for Image Analysis and Knowledge Discovery. Using advanced neurocomputing techniques and a novel knowledge engineering methodology, this thesis attempts to seek some solutions to a set of specific problems that exist within the horticultural domain. In particular it describes a methodology based on previous research into neuro-fuzzy systems for knowledge acquisition, manipulation, and extraction and furthers this area by introducing a novel and innovative knowledge-based architecture for knowledge-discovery using an on-line/real-time incremental learning system based on the Evolving Connectionist System (ECOS) paradigm known as the Evolving Fuzzy Neural Network (EFuNN). The emphases of this work highlights knowledge discovery from these data sets using a novel rule insertion and rule extraction method. The advantage of this method is that it can operate on data sets of limited sizes. This method can be used to validate the results produced by the EFuNN and also allow for greater insight into what aspects of the collected data contribute to the development of high quality produce.New Zealand Foundation for Research Science and TechnologyUnpublishedAgrawal, R., Mannila, H., Srikant, R., Toivonen, H. and Verkamo, A. I. (1996). “Fast discovery of association rules” Advances in knowledge discovery and data mining. American Association for Artificial Intelligence Menlo Park, CA, USA pp. 307–328. Aha, D. W., Kilber, D. and Albert, M. K. (1995). “Instance-based learning algorithms” Machine Learning. 6(1): 37–66. Ahrns, I., Bruske, J. and Sommerl, G. (1995). “On-Line Learning with Dynamic Cell Structure” In F. Fogelman-Soulié and P. Gallinari (eds), Proceedings of the 1995 International Conference on Artificial Neural Networks (ICANN’1995). Vol. II EC2 Nanterre, France pp. 141–146. Amari, S. and Kasabov, N. (eds) (1998). Brain-like Computing and Intelligent Information Systems. Springer-Verlag: Singapore. Andrews, R., Deidrich, J. and Tickle, A. B. (1995). “Survey and critique and techniques for extracting rules from trained artificial neural networks” Knowledge-Based Systems. 8(6): 373–389. Andrews, R. and Geva, S. (2002). “Rule extraction from local cluster neural nets” Neurocomputing. 47(1-4): 277–297. Antani, S., Kasturi, R. and Jain, R. (2002). “A survey on the use of pattern recognition methods for abstraction, indexing and retrieval of images and video” Pattern Recognition. 35(4): 945–965. Baesens, B., Setiono, R., Mues, C. and Vanthienen, J. (2003). “Using neural network rule extraction and decision tables as management science tools for credit-risk evaluation” Management Science. 49(3): 312–329. Bakker, B. (2002). “Reinforcement Learning with Long Short-Term Memory” In T. G. Dietterich, S. Becker and Z. Ghahramani (eds), Advances in Neural Information Processing Systems 14. The MIT Press: Cambridge, MA. Baraldi, A. and Blonda, P. (1999). “A survey of fuzzy clustering algorithms for pattern recognition. II” IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics). 29(6): 786–801. Ben-Hur, A., Horn, D., Siegelmann, H. T. and Vapnik, V. (2001). “A Support Vector Method for Clustering” In T. K. Leen, T. G. Dietterich and V. Tresp (eds), Advances in Neural Information Processing Systems 13. The MIT Press: Cambridge, MA. pp. 367–373. Berkhin, P. (2002). “Survey of Clustering Data Mining Techniques” Technical report. Accrue Software San Jose, CA. Bezdek, J. E. (1993). “Editorial: fuzzy models-what are they, and why?” IEEE Transactions on Fuzzy Systems. 1(1): 1–6. Bilgiç, T. and Türk¸ sen, I. B. (1999). “Measurement of Membership functions: Theoretical and Experimental Work” In D. Dubois and H. Prade (eds), Handbook of Fuzzy Sets and Systems, Vol. 1 Fundamentals of Fuzzy Sets. Kluwer Academic Publishers. pp. 195–202. Billard, A. and Hayes, G. (1999). “DRAMA, a connectionist architecture for control and learning in autonomous robots” Adaptive Behavior Journal. 7(1): 35–64. Birmingham, W. P. and Kim, J. H. (1985). “DAS/Logic: A Rule-based Logic Design Assistant” Proceedings of The Second Conference on Artificial Intelligence Applications. IEEE Computer Society Press. pp. 264–268. Blackmore, J. and Miikkulainen, R. (1993). “Incremental Grid Growing: Encoding High-Dimensional Structure into a Two-Dimensional Feature Map” Proceedings of the 1993 IEEE International Conference on Neural Networks (ICNN’93). Vol. I IEEE Service Center. pp. 450–455. Blue, J. L., Candela, G. T., Grother, P. J., Chellappa, R. and Wilson, C. L. (1994). “Evaluation of Pattern Classifiers for Fingerprint and OCR Applications” Pattern Recognition. 27(4): 485–501. Bonarini, A. (1996). “Evolutionary Learning of Fuzzy rules: competition and cooperation” In W. Pedrycz (ed.), Fuzzy Modelling: Paradigms and Practice. Kluwer Academic Press: Norwell, MA. pp. 265–284. Bourke, P. (2001). “Bicubic Interpolation for Image Scaling”. URL: http://astronomy.swin.edu.au/~pbourke/colour/bicubic/ Brigham, E. (1988). Fast Fourier Transform and Its Applications. Prentice Hall: Upper Saddle River, NJ. Bright, K. P., Sherlock, R. A., Lie, J. and Watsney, M. E. (2000). “Development and use of a whole from model for dairying” In S. Halloy and T. Williams (eds), Applied Complexity: From neural nets to managed landscapes. Institute for Crop and Food Research. pp. 382–389. Brown, D., Craw, I. and Lewthwaite, J. (2001). “A SOM based approach to skin detection with application in real time systems” Proceedings of the British Machine Vision Conference. BMVA. pp. 491–500. Buchanan, B. G. and Shortliffe, E. H. (1984). Rule-based Expert Systems. Addison-Wesley: Reading, MA. Buchanan, J. and Bilkey, D. K. (1997). “Transfer of epileptogenesis between perirhinal cortex and amygdala induced by electical kindling” Brain Research. 771: 71–79. Caron, J. R., Jarvenpa, S. L. and Stoddard, D. B. (1994). “Business Reengineering at CIGNA Corporation: Experiences and Lessons Learned From the First Five Years” Management Information Systems Quarterly. 18(3): 233–250. Carpenter, G. A. and Grossberg, S. (1987). “Discovering order in chaos: Stable self-organization of neural recognition codes” Annals of the New York Academy of Sciences. 504: 33–51. Carpenter, G. A., Grossberg, S., Markuzon, N., Reynolds, J. H. and Rosen, D. B. (1992). “Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps” IEEE Transactions on Neural Networks. 3(5): 618–713. Carpenter, G. and Grossberg, S. (1991). Pattern Recognition By Self Organizing Neural Networks. first edn. The MIT Press: Cambridge, MA. Castellano, G., Castiello, C., Fanelli, A. M. and Mencar, C. (2003). “Discovering prediction rules by a neuro-fuzzy modeling framework” In V. Palade, R. J. Howlet and L. Jain (eds), Lecture Notes in Artificial Intelligence. Vol. 2773 of Knowledge-Based Intelligent Information Engineering Systems. pp. 1242–1248. Chakrabarti, K., Garofalakis, M., Rastogi, R. and Shim, K. (2001). “Approximate query processing using wavelets” VLDB Journal: Very Large Data Bases. 10(2–3): 199–223. Cohen, I. and Berdugo, B. (2001). “Spectral Enhancement By Tracking Speech Presence Probability In Subbands” Proceedings of the IEEE Workshop on Hands Free Speech Communication (HSC’01). IEEE Signal Processing Society. pp. 95–98. Cortes, C. and Vapnik, V. (1995). “Support-Vector Networks” Machine Learning. 20(3): 273–297. Costa, M. A., Braga, A. P. and de Menezes, B. R. (2002). “Constructive and Pruning Methods for Neural Network Design” SBRN ’02: Proceedings of the VII Brazilian Symposium on Neural Networks (SBRN’02). IEEE Computer Society Washington, DC, USA p. 49. Craven, M. (1996). Extracting Comprehensible Models from Trained Neural Networks. PhD thesis Department of Computer Sciences, University of Wisconsin-Madison. Craven, M. W. and Shavlik, J. W. (1997). “Using Neural Networks for Data Mining” Future Generation Computer Systems. 13(2–3): 211–229. Cristea, A. I., Cristea, P. D. and Okamoto, T. (1997). “Neural Network Knowledge Extraction” Rev. Roumaine des Sciences Technique. 42(4): 477–491. Cruces, S., Cichocki, A. and Castedo, L. (2000). “An iterative inversion approach to blind source separation” IEEE Transactions on Neural Networks. 11(6): 1423–1437. Dasarathy, B. V. (1980). “Nosing Around the Neighborhood: A New System Structure and Classification Rule for Recognition in Partially Exposed Environments” IEEE Transactions on Pattern Analysis and Machine Intelligence. 2(1): 67–71. Daubechies, I. (1988). “Orthonormal bases of compactly supporting wavelets” Communications on Pure and Applied Mathematics. 41(7): 909–996. Daubechies, I. (1990). “The Wavelet Transform, Time-Frequency Localization and Signal Analysis” IEEE Transactions on Information Theory. 36(5): 961–1005. Degaudenzi, M. E. and Arizmendi, C. M. (2000). “Wavelet-Based Fractal Analysis of Electrical Power Demand” Fractals. 8(3): 239–245. DeJesus, E. J. and Callan, J. P. (1985). “PEARL: A KNOWLEDGE BASED EXPERT ASSISTED CAD TOOL” Proceedings of the Second Conference on Artificial Intelligence Applications. IEEE Computer Society Press. pp. 258–263. Demuth, H. and Beale, M. (2001). Neural Network Toolbox. 7th edn. The Mathworks, Inc.: Natick, MA. Deng, D. and Kasabov, N. (1999). “Evolving self-organizing map and its application in generation of a world macroeconomic map” In K. Ko and N. Kasabov (eds), Proceedings of the ICONIP/ANZIIS/ANNES’99 International Workshop. University of Otago Press. pp. 7–12. Deng, D. and Kasabov, N. (2000). “ESOM: An Algorithm to Evolve Self-Organizing Maps from On-Line Data Streams” In S.-I. Amari, C. L. Giles, M. Gori and V. Piuri (eds), Proceedings of the IJCNN’2000 on Neural Networks Neural Computing: New Challenges and Perspectives for the New Millennium, Como, Italy. Vol. VI IEEE Computer Society: Los Almitos, CA. pp. 3–8. Deng, D., Koprinska, I. and Kasabov, N. (1999). “RICBIS: New Zealand Repository for Intelligent Connectionist-Based Information Systems” In K. Ko and N. Kasabov (eds), Proceedings of the ICONIP/ANZIIS/ANNES’99 International Workshop. University of Otago Press. pp. 182–185. Deng, Y. and Manjunath, B. S. (2001). “Unsupervised Segmentation of Color-Texture Regions in Images and Video” IEEE Transactions on Pattern Analysis and Machine Intelligence. 23(8): 800–810. Domeshek, E. A., Herndon, M. F., Bennet, A. W. and Kolodner, J. L. (1994). “A Case-Based Design Aid for Conceptual Design of Aircraft Subsystems” Proceedings of the Tenth Conference on AI for Applications. IEEE Press. pp. 63–69. Duin, R. P. W. (2000). PRTools Version 3.0: A Matlab Toolbox for Pattern Recognition. Delft University of Technology. Durand, S. and Froment, J. (2001). “Artifact Free Signal Denoising With Wavelets” Proceedings of ICASSP 2001, IEEE 26th International Conference on Acoustics, Speech, and Signal Processing. IEEE Press. pp. 7–11. Ewing, R. M. and Cherry, J. M. (2001). “Visualisation of expression clusters using Sammon’s non-linear mapping” Bioinformatics. 17(7): 658–659. Faravelli, L. and Yao, T. (1995). “Self-learning fuzzy control of civil structures” Proceedings of the 3rd International Symposium on Uncertainty Modeling and Analysis (ISUMA’95). IEEE Computer Society. pp. 52–56. Fedra, K. and Winkelbauer, L. (2002). “A hybrid expert system, GIS and simulation modeling for environmental and technological risk management” Computer-Aided Civil and Infrastructure Engineering. 17: 131–146. Fisher, R. A. (1936). “The use of multiple measurements in taxonomic problems” Annual Eugenics. 7(II): 179–188. Forgy, C. (1981). “The OPS5 users manual” Technical Report CS-CMU 81-135. Computer Science Department, Carnegie-Mellon University. Frasconi, P., Gori, M. and Soda, G. (1995). “Recurrent Neural Networks and Prior Knowledge for Sequence Processing: A Constrained Nondeterminstic Approach” Knowledge-Based Systems. 8(6): 313–332. Friedman-Hill, E. J. (1997). “Jess, The Expert System Shell for the Java Platform” Technical Report SAND98-8206 (revised). Distributed Computing Systems, Sandia National Laboratories, Livermore, CA. Fritzke, B. (1993). “Growing Cell Structures – a Self-Organizing Network in k Dimensions” In I. Aleksander and J. Taylor (eds), Artificial Neural Networks, 2. Vol. II North-Holland: Amsterdam, Netherlands. pp. 1051–1056. Fritzke, B. (1995). “A growing neural gas network learns topologies” In G. Tesauro, D. S. Touretzky and T. K. Leen (eds), Advances in Neural Information Processing Systems 7. The MIT Press: Cambridge, MA. pp. 625–632. Fu, L. (1989). “Integration of neural heuristics into knowledge-based inference” Connection Science. 1(3): 325–340. Fu, L. (1999). “Knowledge discovery based on neural networks” Communications of the ACM. 42(11): 47–50. Gadwal, D., Greer, J. E. and McCalla, G. I. (1991). “UMRAO: A Chess Endgame Tutor” In J. Mylopoulos and R. Reiter (eds), Proceedings on the 12th International Joint Conference on Artificial Intelligence, (IJCAI’1991). Vol. 2 Morgan–Kaufmann: San Mateo, CA. pp. 1081–1086. Gallant, S. (ed.) (1993). Neural Network Learning and Expert Systems. The MIT Press: Cambridge, MA. Garcez, A. S. A., Broda, K. and Gabbay, D. M. (2001). “Symbolic knowledge extraction from trained neural networks: A sound approach” Artificial Intelligence. 125(1-2): 155–207. García-Pérez, L., Marchant, J., T, Hague and García-Alegre, M. C. (2000). “Fuzzy Decision System for Threshold Selection to Cluster Cauliflower Plant Blobs from Field Visual Images” Proceedings of The 4th World Multiconference on Systemics, Cybernetics and Informatics (SCI’2000). IEEE Press. pp. 23–28. Gasching, J., Klahr, P., Pople, H., Shortliffe, E. and Terry, A. (1983). “Evaluation of Expert Systems: Issues and Case Studies” In F. HayesRoth, D. A. Waterman and D. B. Lenat (eds), Building Expert Systems. Addison-Wesley: Reading MA. pp. 241–280. Geladi, P. and Dåbakk, E. (1995). “An overview of chemometrics applications in near infrared spectrometry” Journal of Near Infrared Spectroscopy. 3(2): 119–132. Gerevini, A., Perini, A. and Ricci, F. (1991). “POMI: A System for Integrated Pest Management” Workshops Notes, AAAI-91 workshop, AI in Natural and Agricultural Resource Management. Gerevini, A., Perini, A. and Ricci, F. (1992). “A Knowledge-Based System for Integrated Pest Management” Proceedings of the Twelfth International Conference on Expert Systems and their Applications. The MIT Press: Cambridge, MA. pp. 749–759. Giles, C. L., Lawrence, S. and Tsoi, A. C. (2001). “Noisy Time Series Prediction Using a Recurrent Neural Network and Grammatical Inference” Machine Learning. 44(1/2): 161–183. Goldenberg, R., Kimmel, R., Rivlin, E. and Rudzsky, M. (2002). “Cortex Segmentation: A Fast Variational Geometric Approach” IEEE Transactions on Medical Imaging. 21(2): 1544–1551. Gonzalez, R. E. and Woods, R. E. (1993). Digital Image Processing. Addison-Wesley. Grossberg, S. (1980). “Biological competition: Decision rules, pattern formation, and oscillations” Proceedings of the National Academy of Sciences. 77(4): 2338–2342. Grosso, W. E., Eriksson, H., Fergerson, R. W., Gennari, J. H., Tu, S. W. and Musen, M. M. (1999). “Knowledge Modeling at the Millennium – The Design and Evolution of Protege-2000” Proceedings of the 12th International Workshop on Knowledge Acquisition, Modeling and Mangement (KAW’99), Banff, Canada, October 1999. University of Calgary. pp. 1–36. Hamker, F. H. (2001). “Life-long learning Cell StructuresÐcontinuously learning without catastrophic interference” Neural Networks. 14: 551–573. Hamza, A. B. and Krim, H. (2001). “Image denoising: A nonlinear robust statistical approach” IEEE Transactions on Signal Processing. 49(12): 3045–3054. Harmon, M. E. and Harmon, S. S. (1996). “Reinforcement Learning: A Tutorial”. URL: http://www.nbu.bg/cogs/events/2000/Readings/Petrov/rltutorial.pdf Hebb, D. (1949). The Organization of Behavior: A Neuropsychological Theory. Wiley: New York. Herrero, J., Valencia, A. and Dopazo, J. (2001). “A hierarchical unsupervised growing neural network for clustering gene expression patterns” Bioinformatics. 17(2): 126–139. Heskes, T. and Kappen, B. (1993). “On-Line Learning Processes in Artificial Neural Networks” In J. Taylor (ed.), Mathematical Foundations of Neural Networks. Elsevier: Amsterdam, Netherlands. pp. 199–233. Hess, C. M. and Kemerer, C. F. (1994). “Computerized Loan Origination Systems: An Industry Case Study of the Electronic Markets Hypothesis” Management Information Systems Quarterly. 18(3): 251–275. Hilton, M. L., Jawerth, B. D. and Sengupta, A. (1994). “Compressing Still and Moving Images with Wavelets” Multimedia Systems. 2(3): 218–227. Hinton, G. (1989). “Connectionist learning procedures” Artificial Intelligence. 40(1–3): 185–234. Hojjatoleslami, A., Sardo, L. and Kittler, J. (1997). “An RBF Based Classifier for the Detection of Microcalcifications in Mammograms with Outlier Rejection Capability” Proceedings of the 1997 International Conference in Neural Networks (ICNN’1997). IEEE Press. pp. 1379–1384. Hoyer, P. O. and Hyvärinen, A. (2000). “Independent Component Analysis Applied to Feature Extraction from Colour and Stereo Images” Network: Computation in Neural Systems. 11(3): 191–210. Hughes, N. P. and Lowe, D. (2003). “Artefactual structure from least squares multidimensional scaling” In S. Becker, S. Thrun and K. Obermayer (eds), Advances in Neural Information Processing Systems 15. The MIT Press: Cambridge, MA. pp. 913–920. Humphrys, M. (1997). Action Selection methods using Reinforcement Learning. PhD thesis Trinity Hall, Cambridge. Iliev, G. and Kasabov, N. (2000). “Tracking of Narrow Band Signals Using Constrained Adaptive Second-Order Filters” Proceedings of the 2000 Conference on Neural Information Pocessing and Intelligent Information Systems, (ICONIP’2000). Vol. 2 IEEE Press. pp. 1367–1370. Ishikawa, M. (1996). “Structural learning with forgetting” Neural Networks. 9(3): 501–521. Itti, L., Chang, L. and Ernst, T. (1998). “Manual and Automatic Extraction of White-Matter Lesions in FLAIR Images” Proceedings of the 6th Annual Meeting of the International Society for Magnetic Resonance in Medicine (ISMRM’1998), Sydney, Australia. ISMRM. p. 2073. Iwai, K., Sekitani, M., Nomura, Y., Ueda, I. and Mizoguchi, R. (1994). “A new Method to Support Designing New Drugs” In Z. Shi (ed.), Proceedings of the 3rd Pacific Rim International Conference on Artificial Intelligence. International Academic Publishers: Beijing, China. pp. 1014–1020. Iyatomi, H. and Hagiwara, M. (2003). “Knowledge extraction from scenery images and recognition using fuzzy inference neural networks” Electronics and Communications in Japan (Part II: Electronics). 86(3): 82–90. Jacobs, C. E., Finkelstein, A. and Salesin, D. H. (1995). “Fast multiresolution image querying” Proceedings of the 22nd annual conference on Computer graphics and interactive techniques (SIGGRAPH’95). ACM Press. pp. 277–286. Jang, J. S. R. and Sun, C. T. (1997). Neuro-Fuzzy and Soft Computing: A Computational Approach to Learning and Machine Intelligence. Prentice Hall: Upper Saddle River, NJ. Jang, R. (1993). “ANFIS: Adaptive Network-Based Fuzzy Inference System” IEEE Transactions on System, Man, and Cybernetics. 23(3): 665–685. Jang, R. (2001). Fuzzy Logic Toolbox. 2nd edn. The Mathworks, Inc.: Natick, MA. Jeffrey, N., Dillon, W. P. and Twichell, D. (1999). “Seismic reflection results from the Gyre 1997 cruise at the Bryant Canyon of the Louisiana gulf coast” U.S. Geological Survey Open-File Report 99–449. U.S. Geological Survey Woods Hole, MA. Jervis, J. (2002). “Lecture 5 notes, Multimedia Image Design”. URL: http://www.it.bond.edu.au/inft140/Lectures061/Lecture5/Lecture%205.htm Jin, Y. and Sendhoff, B. (2003). “Extracting interpretable fuzzy rules from RBF networks” Neural Processing Letters. 17(2): 149–164. Jordan, M. I. and Jacobs, R. A. (1994). “Hierarchical Mixtures of Experts and the EM Algorithm” Neural Computation. 6: 181–214. Kaelbling, L. P., Littman, M. L. and Moore, A. W. (1996). “Reinforcement Learning: A Survey” Journal of Artificial Intelligence Research. 4: 237–285. Kandel, A. (1992). Fuzzy Expert Systems. CRC Press: Boca Raton, FL. Kantrowitz, M., Horstkotte, E. and Joslyn, C. (2003). “Answers to Frequently Asked Questions about Fuzzy Logic and Fuzzy Expert Systems” comp.ai.fuzzy. pp. 1–20. URL: ftp.cs.cmu.edu:/user/ai/pubs/faqs/fuzzy/fuzzy.faq Kasabov, N. (1996a). Foundations of Neural Networks, Fuzzy Systems and Knowledge Engineering. first edn. The MIT Press: Cambridge, MA. Kasabov, N. (1996b). “Learning fuzzy rules and approximate reasoning in fuzzy neural networks and hybrid systems” Fuzzy Sets and Systems. 82(2): 2–20. Kasabov, N. (1996c). “The ECOS Framework and the ECO Learning Method for Evolving Connectionist Systems” Journal of Advanced Computational Intelligence. 2(6): 195–202. Kasabov, N. (1997). “Fuzzy rules extraction, reasoning and rules adaptation in fuzzy neural networks” Proceedi