Jie shi xing qu dong de zhi neng ruan jian ke kao xing gong cheng

The reliability of software, including traditional software and intelligent software, is crucial to both end-users and service providers. However, the increasing scale and complexity of traditional software and the massive parameters of intelligent software, make both software hard to understand, which poses great challenges for software reliability engineering. Especially, detecting and tracking the problems becomes inefficient and error-prone when unexpected software behaviors occur. In this thesis, we study the intelligent software reliability engineering from the interpretability perspective. Specifically, we propose to interpret traditional software with logs and intelligent software with gradients and bilingual knowledge.Firstly, we conduct an experience report on log-based anomaly detection, which uncovers abnormal software behaviors using the interpretable logs. The anomaly detection plays a vital role in traditional software reliability engineering. Although anomaly detection has been widely studied in recent years, a comprehensive benchmark and an open-source toolkit are lacking. We implement six representative anomaly detection methods and evaluate their performance in terms of effectiveness and efficiency. We obtain five insightful findings and make these methods open-source for easy reuse and further study.Secondly, we propose an intelligent log-based method to identify impactful problems in software systems. There could be various types of problems, while some are more impactful, leading to the degradation of Key Performance Indicator (KPI). To tackle the challenges of highly imbalanced log distribution and label scarcity, we propose a novel cascading clustering and KPI correlation method to identify impactful problems. Experimental results on three real-world datasets confirm the effectiveness and efficiency of our proposed method.Thirdly, we focus on interpreting the intelligent software, which is the first step towards its reliability engineering. We take the neural machine translation (NMT) model as our testbed. Although NMT advances the state-of-the-art performance, its unsatisfactory interpretability poses great challenges for model debugging and improving. We propose to understand the model input-output behaviors by exploiting the internal gradients to estimate the word importance. Comprehensive experiments on different perturbation operations, language pairs, and model architectures validate the effectiveness of our method. We also employ the proposed word importance to detect the under-translation error in NMT.Fourthly, we explore the interpretability of NMT models by assessing its learned bilingual knowledge. The NMT model contains tons of uninterpretable parameters, but the essential bilingual knowledge that model embeds for translation is unclear. We propose to quantitatively assess the bilingual knowledge that NMT models learn using the human-understandable phrase table. Extensive experiments on widely-used datasets show that the phrase table is reasonable and consistent. Moreover, we obtain some interesting findings after analyzing the model learning dynamics and some advanced model improvement techniques.In summary, this thesis targets at studying the interpretability-based reliability engineering of both traditional software and intelligent software. Comprehensive experiments on widely-used datasets confirm the effectiveness and efficiency of our proposed methods.軟件，包括傳統軟件和智能軟件，其可靠性對於終端用戶和服務提供方都至關重要。然而，傳統軟件急劇增加的規模和複雜度，以及智能軟件中巨量的參數，使得軟件難以被理解，從而給軟件可靠性工程帶來了巨大的挑戰。尤其當意料之外的軟件行為出現的時候，檢測和追蹤問題變得非常的低效和易錯。在本論文中，我們從可解釋性的角度探究智能軟件可靠性工程。具體來說，我們提出用日誌來解釋傳統軟件以及用梯度和知識來解釋智能軟件。首先，我們基於日誌的異常檢測算法基礎上進行經驗研究。異常檢測在傳統軟件可靠性分析中極為重要，其目的在於利用可解釋性的日誌來找出異常的軟件行為。雖然異常檢測已被廣泛研究，但始終缺乏詳盡的基準探究和開源工具。我們實現了六種代表性的異常檢測方法，並且從有效性和效率的角度對這些方法進行評估。我們總結出五個有趣結論，並將這些方法開源以供後續使用和研究。其次，我們提出一種智能化的，基於日誌的方法來鑒別出軟件系統中有影響力的問題。軟件系統中的問題種類可能有很多，但是有些問題會相較於其餘問題更有影響力，也更容易導致系統性能監控指標的下降。為了解決日誌分佈極度不均衡和標籤缺乏的挑戰，我們提出一種新穎的級聯聚類以及關聯系統性能監控指標的方法來鑒別出有影響力的問題。我們在三個真實數據集上驗證了我們方法的有效性和高效率。再其次，我們關注與解釋智能軟件的行為，該可解釋性是邁向智能軟件可靠性工程的第一步。在本論文中，我們將神經機器翻譯模型作為我們的研究對象。雖然神經機器翻譯模型已經極大地促進了機器翻譯任務的效果，它的不可解釋性卻阻礙了模型的糾錯和提升。我們提出用模型內部的梯度信息來估計詞重要性，從而理解模型的輸入-輸出行為。我們在不同擾動，語言對和模型架構上實驗證明我們方法的有效性。我們還將這種基於梯度的方法用於檢測神經機器翻譯的漏譯問題。最後，我們從評估模型學習到的雙語知識角度來探究神經機器翻譯模型的可解釋性。神經機器翻譯模型中包含大量不可解釋的參數，並且模型中的翻譯所需雙語知識是不清楚的。我們提出用短語詞表的方法來量化評估模型學習到的雙語知識。我們在常見數據集上進行大量的實驗，結果表明短語表是一種合理且通用的評估方法。除此之外，我們分析了模型的動態學習過程和一些模型改進方法，並從中得到了一些有趣的結論。綜上所述，本論文旨在從可解釋性角度出發研究傳統軟件和智能軟件的可靠性工程。我們在廣泛使用的數據集上進行了大量豐富的實驗，結果證明了我們所提出的這些方法的有效性和高效性。He, Shilin.Ph.D. Chinese University of Hong Kong 2020.Includes bibliographical references (leaves 161-186).Abstracts also in Chinese.Title from PDF title page (viewed on September 13, 2022).He, Shilin