A bilingual speech interface for New Zealand English to Māori

The 'Bilingual Speech Interface for New Zealand English to Māori ' is part of the 'Intelligent Human Computer Interfaces' project under 'Objective 3' of the 'Connectionist-Based Information Systems' programme (FoRST UOO606) Department of Information Science, University of Otago. The project experiments with artificial intelligent knowledge-based engineering methodologies and techniques for designing tools that utilise a hybrid system approach that is both adaptable and flexible to different speakers and languages-namely New Zealand English and Māori. Artificial neural networks, fuzzy rule-based inferencing techniques, genetic algorithms and multimedia based applications form the multiple paradigm approaches for solving real applied problems for speech generation. All are fundamental in the developmental direction of an intelligent human computer interface. Multimedia interfaces to databases use conventional software engineering techniques for the management, access and retrieval of information, therefore, implementing an interface application to access a speech and language database means that a change in computer interaction from manual to automatic control is facilitated between the user and the database. The 'Hybrid Neuro-Fuzzy Speech Recognition System' called HySpeech is based on isolated word recognition for New Zealand English that utilises an English-Māori lexical database to provide an automatic language translator or 'Talking Dictionary'. The current development with version two of HySpeech incorporates the advanced 'Fuzzy Neural Network' models with new 'Learning with Forgetting' algorithms for better speaker adaptive capabilities, an aggregation function provides cleaner phoneme compression and 'Self-Organising Maps' for phoneme and word lists-all to facilitate new language modelling techniques. The language information is housed in a separate database management system designed for HySpeech, it contains; the speakers' characteristics, English and Māori words, phonetic transcriptions and pronunciations, segmental labels, phoneme activation and co-ordinate vectors, and a growing set of digitised Māori speech examples. The interface component of HySpeech thus comprises and text examples from the aforementioned database, interactive dialogues, and a graphical user interface generated environment. Also, experiments in artificial language generation for speech synthesis provide the system with some knowledge and information about the languages. The system could therefore be capable of recreating a form of machine generated speech. Many approaches are available to facilitate synthesised speech, so by utilising the present methodologies and techniques, there is a future solution that can run parallel to the existing direction of HySpeech, for a complete bilingual speech interface. Given that there is presently no operational system that can synthesise New Zealand English and Māori speech, this thesis will also provide some basis to facilitate the further research and development in that area.ndrewes, M. and Tamblyn, R. (1994) On Call™ Innovative Language Learning Systems, Language Learning Centre, University of Otago, Dunedin. Apple Computer Inc. (1991) Inside Macintosh, Volume VI. Addison Wesley. Apple Computer Inc. (1996) Inside Macintosh, On-Line Version. [http://speech.apple.com/dev/insidemac/] Apple Computer Inc. (1997) PlainTalk: An Apple Speech Technology White Paper. [http://speech.apple.com/] Bäckström, C. and Sandewall, E. (1994) Current Trends in AI Planning. EWSP’93 – 2nd European Workshop on Planning, IOS Press, Amsterdam. Barlow, C. (1990a) An Alternative Format for Maori Archival Materials, Archifacts, April 1990, pp. 22-28. Barlow, C. (1990b) Me Ako Taatou i Te Reo Maaori , By Bruce Biggs, Uniprint, Orakei. Barlow, C. (1994) Tikanga Whakaaro: Key concepts in Mäori culture. Oxford University Press, Auckland. Bauer, L. (1986) Notes on New Zealand English Phonetics and Phonology. English World- Wide, Volume 7, No. 2, pp. 255-258. Bauer, L. (1994) Introducing the Wellington Corpus of Written New Zealand English, Te Reo, Journal of the Linguistic Society of New Zealand, Volume 37, University of Auckland. Bauer, W., Parker, W., Evans, T. (1993) Maori Descriptive Grammars, Routledge, London and New York. Bell Laboratories, Lucent Technologies, Murray Hill, New Jersey. [http://www.bell-labs.com/project/tts/] Bender, B. W. (1971) Linguistic Factors in Maori Education, A Report. New Zealand Council for Educational Research, Wellington. Benton, R. A. (1979) Is Bilingualism a Handicap? New Zealand Council for Educational Research, Wellington. Benton, R. A. (1981) The Flight of the Amokura, Oceanic Languages and Formal Education In the South Pacific, New Zealand Council for Educational Research, Wellington. Benton, R. A., Tumoana, H., Robb, A. (1982) Ko Ngä Kupu Pü Noa o Te Reo Mäori: The First Basic Maori Word List, New Zealand Council for Educational Research, Wellington. Benton, R. A. (1982) The Anatomy of a Word List, New Zealand Council for Educational Research, Wellington. Benton, R. A. (1990) The History and Development of the Mäori Language, Dirty Silence Aspects of Language and Literature in New Zealand, Essays arising from the University of Waikato Winter Lecture Series of 1990, Oxford University Press, Auckland. Benton, R. A. (1992) Maori English: A New Zealand Myth?, New Zealand English Newsletter, Number 6, Department of English, University of Canterbury, Christchurch, pp. 27-35. Bezdek, J. and Pal, N. (1993) An Index of Topological Preservation and its Application to Self-Organising Feature Maps. Proceedings of 1993 International Joint Conference on Neural Networks, pp. 2435-2440. Biggs, B. (1981) The Complete English-Maori Dictionary, Auckland University Press, Auckland. Biggs, B. (1996) Lets Learn Maaori. A Guide to the Study of the Maori Language, Reprinted by Uniprint, Auckland. Black, A. W. and Taylor, P. (1994) CHATR: a generic speech synthesis system. COLING-94, volume II, pp. 983-986, Kyoto. Japan. Borland (1992). Paradox for Windows ObjectPAL Reference. Borland International, Inc. Braspenning, P., Taylor, J., Gallinari, P., Kasabov, N. (1990) Connectionism & AI Lectures. ISAI’90-International School in Artificial Intelligence, Albena, Bulgaria. Bregman, A. S. (1994) Auditory Scene Analysis, The Perceptual Organization of Sound. A Bradford Book, The MIT Press, Cambridge. Campbell, N. (1996) CHATR: A High-Definition Speech Re-Sequencing System. Acoustic Society of America & Acoustic Society of Japan, 3rd Joint Meeting, Honolulu, pp. 1223- 1228. Campbell, N. (1997) Processing a Speech Corpus for CHATR Synthesis. ATR Interpreting Telecommunications Research Laboratories, Kyoto. Capron, H. (1990) Computers – Tools for an Information Age, 2nd Edition, Redwood City, Benjamin/Cummings Publishing. Chomsky, N. and Halle, M. (1968) The Sound Pattern of English. New York, Harper and Row. Clark, J. and Yallop, C. (1990) An Introduction to Phonetics and Phonology. Cambridge, Massachusetts, B. Blackwell. Cole, R., Noel, M., Burnett, D. C., Fanty, M., Lander, T., Oshika, B., Sutton, S. (1994) Corpus Development Activities at the Center for Spoken Language Understanding, In Proceedings of the ARPA Workshop on Human Language Technology. Creative Technology Ltd. (1997) Creative TextAssist™. [http://www-nt-ok.creaf.com/techsupt/devcnr/tassist.html] Cruttenden, A. (1994) Gimson’s Pronunciation of English, 5th Edition, Hodder Headline Group, London. Crystal, D. (1980) A First Dictionary of Linguistics and Phonetics. The Language Library, University Press, Cambridge. Crystal, D. (1992) The Cambridge Encyclopedia of Language, Cambridge University Press, Cambridge. Date, C. J. (1990) An Introduction to Database Systems. Volume 1, 5th Edition. Addison- Wesley Publishing Co. Demuth, H. and Beale, M. (1996) Neural Network ToolBox For Use with MatLab. The Math Works, Inc. Natick, Massachusetts. Dutoit, T., V, Pagel., N, Pierret., F, Bataille., O, Van Der Vrecken. (1996) “The MBROLA Project: Towards a Set of High-Quality Speech Synthesisers Free of Use for Non-Commercial Purposes”. Proc ICSLP’96, Philadelphia, Vol. 3, pp. 1393-1396. Fahlman, S. (1988) Faster-Learning Variations on Back-Propagation: An Empirical Study, in Proceedings Connectionist Models Summer School, Morgan Kaufmann. Foster, J. (1993) He Whakamärama A New Course in Mäori, Reed Books, Auckland. Fraser, S. (1995) DictionaryEdit Online User Manual. Centre for Population Biology, Imperial College at Silwood Park, Birkshire. Fromkin, V., Rodman, R., Collins, P., Blair, D. (1988). An Introduction to Language , Second Australian Edition, Holt, Rinehart and Winston, Australia. FuzzyCOPE V1.0, (1995) User Manual, Information Science Department, University of Otago, Dunedin. FuzzyCLIPS Version 6.02, (1994) User’s Guide, Knowledge Systems Laboratory, Institute for Information Technology, National Research Council, Canada. Giegerich, H. (1992) English Phonology: An Introduction. Cambridge University Press, Cambridge. Goodman, D. (1993) The Complete HyperCard 2.2 Handbook. 4th Edition, Random House, New York. Hanks, P. (1980) Meaning and Grammar. Editor, Collins Dictionary of the English Language. An extensive coverage of contemporary international and Australian English. William Collins Sons & Co. Ltd, Sydney. Harawira, K. T; revised by Käretu, T. (1994) Teach Yourself Mäori, Reeds Books, Auckland. Harlow, R. (1987) A Word-List of South Island Maori, Te Reo Monographs, Linguistic Society of New Zealand, University of Auckland. Harlow, R. (1990) A Name and Word Index to Ngä Mahi a Ngä Tüpuna, University of Otago Press, Dunedin. Holland, J. (1994) Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence. MIT Press, CA, MA. Holmes, J. (1990) The Role of the Sociolinguist in Society, Dirty Silence, Aspects of Language and Literature in New Zealand, Essays arising from the University of Waikato Winter Lecture Series of 1990, Oxford University Press, Oxford. Hunt, A. J. and Black, A. W. (1996) Unit Selection in a Concatenative Speech Synthesis System Using a Large Speech Database. Proceedings: ICASSP, Atlanta, GA. Karaali, O., Corrigan, G., Gerson, I. (1996) Speech Synthesis with Neural Networks. World Congress on Neural Networks, San Diego, pp. 45-50. Kärena-Holmes, D. (1993) Grammar Basics English – Mäori, 60 Royal Tce, Dunedin. Kärena-Holmes, D. (1995) Mäori Language Understanding the Grammar, The Printing Department, University of Otago, Dunedin. Kasabov, N., Watson, C., Sinclair, S., Kilgour, R. (1994) Integrating Neural Networks and Fuzzy Systems for Speech Recognition. 5th International Australian Speech Science and Technology Conference, Perth, pp. 462-467. Kasabov, N., Watson, C. (1994) Automatic Speech Recognition: Methods, Tools and Their Application for Communication and Intelligent Information Systems. A Report on a Research Project Funded by Telecom NZ Ltd. Information Science, University of Otago, Dunedin, New Zealand. Kasabov, N., Sinclair, S., Kilgour, R., Watson, C., Laws, M., Kassabova, D. (1995) Intelligent Human Computer Interfaces and the Case Study of Building English-to-Mäori Talking Dictionary, Kasabov, N., Coghil, G. (Eds.), Proceedings ANNES 95’ University of Otago, Los Alamitos, CA: IEEE Computer Society Press, pp. 294-297. Kasabov, N., Purvis, M., Sallis, P. (1996a) Connectionist-Based Information Systems: A Proposed Research Theme, The Information Science Discussion Paper Series, No 96/03, Dunedin, New Zealand. Kasabov, N., Kim, J., Gray, A., Watts, M. (1996b) Fuzzy Neural Network Architecture for Adaptive Learning and Knowledge Acquisition, International Journal of Information Sciences: Applications, North Holland. Kasabov, N. (1996) Foundations of Neural Networks, Fuzzy Systems and Knowledge Engineering, The MIT Press, CA, MA. Kasabov, N. and Gray, A. (1996) Round-trip System Engineering in Neuro-Fuzzy Hybrid Systems. Submitted to Journal of Intelligent & Fuzzy Systems. Kasabov, N. (1997) A Framework for Intelligent ‘Conscious’ Machines Utilising Fuzzy Neural Networks and Spatial temporal Maps and a Case Study of Multilingual Speech Recognition, In: S. Amari and N. Kasabov (eds) “Brain-like Computing and Intelligent Information Systems”, Springer Verlag, Singapore Kasabov, N. and Kozma, R. (1997) Fuzzy Neural Networks and Fuzzy Spatio Temporal Maps, In: N. Kasabov and R. Kozma (eds) Neuro-fuzzy Tools and Techniques, Physica Verlag, Heidelberg. Kasabov, N., Kozma, R., Kilgour, R., Laws, M., Taylor, J., Watts, M., Gray, A. (1997) Neuro-fuzzy Techniques for Speech Data Analysis and Adaptive Speech Recognition, In: N. Kasabov and R. Kozma (eds) Neuro-fuzzy Tools and Techniques, Physica Verlag, Heidelberg Kasabov, N., Kozma, R., Kilgour, R., Laws, M., Taylor, J., Watts, M., Gray, A. (1997a) Methodology for Speech Data Analysis and a Framework for Adaptive Speech Recognition Using Fuzzy Neural Networks. ICONIP/ANZIIS/ANNES`97. vol 2, pp. 1055-1060. Keegan, T. and Treweek, P. (1994) Ngä Tautono Rorohiko Course Manual. Department of Computer Science, University of Waikato, Hamilton. Keegan, P. (1996) Te Wahapu – Mäori/English Lexical Database Web Public Access, New Zealand Council for Educational Research, Wellington. [http://www.dia.govt.nz/dia/general/dictionary/maori/public/wordsearch.html] Keegan, P. (1997) Kimikupu Hou Mäori Lexical Database on the Web: Reflections and Possible Future Directions. In Proceedings NAMMSAT Conference, October 1997, Massey University, Palmerston North. Kilgour, R. “Hybrid Fuzzy Systems and Neural Networks for Speech Recognition.” Unpublished M.Sc. (Cognitive) thesis, University of Otago, 1996. Kilgour, R. (1997) Report on file formats for sound and speech files. Department of Information Science, University of Otago. Knowles, M. (1983) Andragogy: An Emerging Technology for Adult Learning, M. Tight (ed.) In, Adult Learning and Education. London: Open University. 53-70. Kohonen, T. (1990) The Self-Organizing Map, Proceedings of the IEEE, vol.78, N-9, pp. 1464-1479. Kohonen, T. (1997) Self-Organizing Maps, Second Edition, Springer-Verlag, New York. Kosko, B. (1992) Neural Networks and Fuzzy Systems: A Dynamical Approach to Machine Intelligence, Prentice Hall, New Jersey. Kozma, R., Sakuma, M., Yokoyama, Y., Kitamura, M. (1996) On the accuracy of mapping by neural networks trained with backpropagation with forgetting, Neurocomputing, 13, pp. 295- 311. Kuchera, H. and Francis, W. N. (1967) Computational Analysis of Modern-Day American English, Brown University Press, Providence. Kumar, S. R. and Ghoshal, J. (1997) Neuro Fuzzy Approach to Pattern Recognition. Neural Networks, Vol. 10, No. 1, pp. 161-182. Kurzwell, R. (1997) When Will HAL Understand What We Are Saying, In: Stork, D. G (Ed) HAL’S Legacy: 2001’s Computer As Dream and Reality. The MIT Press, Cambridge, Massachusetts. Ladefoged, P. (1993) A Course In Phonetics, Harcourt Brace Jovanovich, Inc. Laws, M. (1995) A Computer Assisted Mäori Language Laboratory. The Design, Development and Implementation of an integrated package of Mäori language course material in a computerised environment—is it possible and practical? Proceedings of the Inaugural NAMMSAT Conference, Auckland University. Laws, M. (1995) Building Speech Interfaces to Databases: A Case Study for an English-to- Mäori Talking Dictionary, Addendum to ANNES 95’, University of Otago, Dunedin, New Zealand. Laws, M. “New Zealand English to Mäori Lexical Database: A Talking Dictionary”. Unpublished Research Dissertation, University of Otago, 1995. Laws, M. (1996) The ongoing development of an Intelligent Human Computer Interface to a New Zealand English-Mäori Speech Database: A Talking Dictionary. In Proceedings, NAMMSAT Conference, Hamilton. Laws, M. (1997) Integrating Text and Speech into Databases, Information Systems and Knowledge Engineering for Human Computer Interaction. The progressive development of an Integrated Bilingual Interface. In Proceedings NAMMSAT Conference, October 1997, Massey University, Palmerston North. Lewis, G. (1996) The Origins of New Zealand English: A Report on Work in Progress. The New Zealand English Journal 10, pp. 25-30. Lewis, K. R., Pettey, M., Shneiderman, B. (1993) Speech versus mouse commands for word processing: an empirical evaluation. International Journal of Man-Machine Studies, 39, pp. 667-687. Linguistic Data Consortium. (1996) LDC Catalog, University of Pennsylvania, Philadelphia. Linggard, R. (1994) Speech Science and Technology - Review and Perspective. 5th International Australian Speech Science and Technology Conference, Perth, pp. 576-578. Luce, P., Pisoni, D., Goldinger, S. (1990) Similarity Neighborhoods of Spoken Words. Cognitive Models of Speech Processing: Psycholinguistic and Computational Perspectives. Ed G. Altmann, The MIT Press, Massachusetts. Maclagan, M (1987) An acoustic study of New Zealand Vowels. New Zealand Speech Therapists' Journal Vol. 37, No 1, pp. 20-26. Maclagan, M. and Gordon, E. (1995) The Changing Sound of New Zealand English. The New Zealand Speech-Language Therapists' Journal Vol. 50, pp. 32-40. McCrone, J. (1993) Computers that Listen. New Scientist, 4th December 1993, pp. 30-35. McCulloch, N., Bedworth, M., Brible, J. (1987) NETspeak—A re-implementation of NETtalk. Computer Speech and Language, Vol 2, pp. 289-301. McGregor, G. and Williams M. (1990) (Eds.) Dirty Silence, Aspects of Language and Literature in New Zealand, Essays arising from the University of Waikato Winter Lecture Series of 1990, Oxford University Press, Oxford. Malfrère, F. and Dutoit, T. (1996) High-Quality Speech Synthesis For Phonetic Speech Segmentation. Circuits Theory and Signal Processing Lab, Faculté Polytechnique de Mons, Belgium. Mäori Language Commission. (1992) Te Matatiki – Ngä Kupu Hou a Te Taura Whiri i te Reo Mäori, Mäori Language Commission, Wellington. Mäori Language Commission. (1996) Te Matatiki – Ngä Kupu Hou a Te Taura Whiri i te Reo Mäori . Oxford Press, Auckland. Meisel, W. S. (1993) Talk to Your Computer: Voice technology lets you verbally command your computer or convert speech to text. BYTE October 1993, pp. 113-120. Microsoft (1994) Access® Version 2.0. Relational Database Management System for Windows™, Building Applications, Microsoft Corporation. Microsoft (1995)Building Applications with Microsoft Access for Windows 95. Programming with Visual Basic for Applications, Microsoft Corporation, Australia. Mitra, S. and Pal, S. K. (1995) Fuzzy Multi-Layer Perceptron, Inferencing and Rule Generation. IEEE Transactions on Neural Networks, Vol. 6, No. 1, pp. 51-63. Möbius, B., Schroeter, J., Santen, J., Sproat, R., Olive, J. (1996) Recent Advances in Multilingual Text-To-Speech Synthesis. AT&T Bell Laboratories, Murray Hill, New Jersey. Mohri, M., Riley, M., Sproat, R. (1996) Algorithms for Speech Recognition and Language Processing. A Tutorial presented at COLING’96 (Computational Linguistics), Copenhagen. Molich, R. and Nielsen, J. (1990) Improving a Human-Computer Dialogue, Communications of the ACM, Vol 33, No 3, March, pp. 338-343. Moorfield, J. C. (1989) Te Käkano, Longman Paul Ltd, Auckland. Moorfield, J. C. (1988) Te Whanake 1 Te Käkano: Pukapuka ärahi i te kaiwhakaako, Te Whare Wänanga o Waikato. Morgan, D. and Scofield, C. (1991) Neural Networks and Speech Processing. Kluwer Academic Publisher, Massachusetts. Ngata, H. M. (1993) English-Mäori Dictionary, Learning Media, Wellington. Olive, J. P. (1997) “The Talking Computer”, In: Stork, D. G (Ed) HAL’S Legacy: 2001’s Computer As Dream and Reality. The MIT Press, Cambridge, Massachusetts. Owens, F. (1994) Signal Processing of Speech. The MacMillan, McGraw-Hill, Inc. Pagel, V. “New Database” Personal email communication. 10 June 1997. Faculté Polytechnique de Mons, Belgium. [pagel@tcts.fpms.ac.be] Rabiner, L. (1994) Applications of Voice Processing to Telecommunications, Proceedings of the IEEE, Vol 82, No. 2, Feb, pp. 199-288. Rheingold, H. (1992) Virtual Reality, Mandarin Paperbacks, London. Ritter, H. and Kohonen, T. (1989) Self-Organising Semantic Maps. Biological Cybernetics vol 61, Springer-Verlag, pp. 241-254. Rob, P. and Williams, T. R. (1995) Database Design and Applications Development with Microsoft Access 2.0. McGraw-Hill, San Francisco. Ryan, P. M. (1985) The Revised Dictionary of Modern Mäori, Heinemann Publishers. Ryan, P. M. (1990) The Revised Dictionary of Modern Mäori, Heinemann Publishers. Sejnowski, T. J. and Rosenberg, C. (1986) NetTalk: A Parallel Network that Learns to Read Aloud, Johns Hopkins University Electrical and Computer Science Technical Report, Johns Hopkins University, Baltimore. Sejnowski, T. J. and Rosenberg, C. (1988) NetTalk Corpus, Johns Hopkins University, Cognitive Science Center, Baltimore. Sinclair, S. and Watson, C. (1995) The Development of the Otago Speech Database. In Proceedings ANNES 95’, University of Otago, Dunedin. Sinclair, S. “Development of an Isolated Speech Digit Recognition System Based on Back Propagation Neural Networks.” Unpublished M.Com. thesis, University of Otago, 1996. Sproat, R. (1996) Multilingual Text Analysis for Text-to-Speech Synthesis. 12th European Conference on Artificial Intelligence, Edited by W. Wahlster, Sproat, R. and Shih, C. “Maori Info.” Personal email communication. 27 May 1997. Bell Laboratories, Lucent Technologies, Murray Hill. [rws@research.bell-labs.com] Stork, D. G (1997) HAL’S Legacy: 2001’s Computer As Dream and Reality. The MIT Press, Cambridge, Massachusetts. Sullivan, K. P. H. and Damper, R. I. (1995) Novel-word pronunciation: A cross-language study. Artificial Intelligence Laboratory, Computer & Information Science, University of Otago, Dunedin. Sullivan, K. P. H. (1995) Text-to-speech conversion fo