An OCR Classifier for Republican Chinese Newspaper Text – Creation and Post-processing

For more than a decade, Republican magazines and newspapers have been collected by institutes and projects now joined in the Centre for Asian and Transcultural Studies (CATS) at Heidelberg University. Our platform “Early Chinese Periodicals Online” (ECPO, https://uni-heidelberg.de/ecpo), provides open access to more than 300.000 digital images and their metadata, cf. [6] and [7]. Since the material consists mostly of image scans, the project ran a number of experiments to explore possible approaches towards full text generation [8]. For newspapers printed in Latin scripts much has changed since Rose Holley deemed the use of “’training’” facility (artificial intelligence) in the OCR software” as “Not viable for cost effective mass scale digitization” and noted “Do not pursue” in her list “Potential methods of improving OCR accuracy suggested by ANDP team” [9, table 2, item 9]. Today, when Liebl and Burghardt write that “transforming [historical newspapers] into machine-readable data by means of OCR poses some major challenges” they do that while they introduce their own OCR pipeline [9]. Unfortunately, these approaches cannot just be adopted to historical Chinese newspapers. As we have shown, especially complex layout and resulting difficulties of reliable automatic page segmentation have so far prevented full text generation of these newspapers even within China [8], [11], [12]. In our paper we present the first results of a systematic approach towards full text extraction from a Republican China newspaper (Fig. 1). Our basis is a small corpus for which text ground truth exists. We present our character segmentation method which produces about 90.000 images of characters. Based on the hypothesis that pre-training on extensive amounts of suitably augmented character images will increase the OCR accuracy for evaluation on real-life character image data, we generate synthetic training data. We then compare the OCR recognition results and show that a combination of synthetic and real characters produces best results. Finally, we propose a method that makes use of a masked language model for OCR error correction