SCHOLIA-An XAI Framework for APR

Nethum Lamahewage, Nimantha Cooray, Ridwan Shariffdeen, Sandareka Wickramanayake, Nisansa de Silva
IEEE/ACM International Workshop on Automated Program Repair (APR)

Automated Program Repair (APR) can assist developers by automatically generating patches for buggy code. However, as recent techniques leverage deep learning models, developers do not know why the model generated a particular patch. Existing Explainable AI (XAI) techniques, such as SHAP, can be applied to APR, however, their complexity raises questions about whether developers find such explanations understandable. In this work, we develop a novel framework SCHOLIA, with two extensions to feature attribution methods to make them more understandable to the developers. First generates a text explanation based on attribution scores. Second creates a visualization capturing the transformation of the patch based on the impact of code tokens, named patch transformation. We evaluated the proposed new two explanations types compared to SHAP, using a user survey. The survey received responses from 106 participants. Accordingly, 68.9\% (P < .05) and 64.2\% (P < .05) of participants agreed that text explanation and patch transformation methods are easy to understand, while only 17.9\% (P < .05) agreed with the original SHAP explanation. The survey responses indicate, with statistical significance, that our extensions to SHAP are easier to understand than the original SHAP explanations

Keywords: Machine Learning / Deep Learning | Automated Program Repair | Explainable Artificial Intelligence |