Faithfulness of explanations in deep reinforcement learning

Neural Networks (NNs) are often cited as black-boxes, i.e., models which are not interpretable by human cognition and whose function learnt from the data is not directly accessible. For this reason, several techniques for trying to probe into one or more aspects of the inner functioning of NNs have been proposed, and this field has been dubbed Explainable AI (XAI). Input Attribution (IA) methods are tools to approximate which parts of an input are important in determining the model prediction.

NNs can also be employed as controllers of Reinforcement Learning (RL)-based agents, whereas the model is used to learn the best policy for navigating an environment in a given state. In this sense, IA methods, or any other explainability technique, can be a great tool for getting intuitions about the reasons why an agent executes a specific action. The field that looks for explainability in RL agent behaviors is called explainable RL (XRL). The goal of XRL is to elucidate the decision-making process of learning agents in sequential decision-making settings.

One of the most pressing challenges of XAI is connected to the assessment of the quality of the explanations provided: in fact, many XAI tools are mere approximations of the underlying decision process operated by the NN, and the explanations can be widely inaccurate in that regard. A straightforward way of evaluating the explanations is to consider the faithfulness of the explanation: for instance, we could ask ourselves whether the input parts identified via IA are actually important for the NN—by perturbing or masking these areas, one reasonably expects the action of the underlying RL agent to change.