Task-agnostic exploration for sparse reward environments

Exploration is a fundamental aspect of reinforcement learning (RL). Its effectiveness plays a crucial role in the performance of RL algorithms, particularly when confronted with sparse extrinsic rewards. In reinforcement learning (RL), exploration is generally divided into two settings. The first, task-driven exploration, involves a well-defined reward where the agent aims to explore to maximize long-term rewards. However, in real-life scenarios, external rewards are often sparse or completely unknown [1]. The second, task-agnostic exploration, occurs when an agent must explore a new environment without the guidance of any external reward

[1] describes a setup where the agent first learns to explore multiple environments in a task-agnostic manner without any extrinsic goal. Subsequently, the agent efficiently transfers this learned exploration policy to more effectively explore new environments while solving tasks. The researchers introduced the Change-Based Exploration Transfer (C-BET) algorithm, which merges exploration with both extrinsic and intrinsic goals. The C-BET algorithm can be integrated into various RL algorithms. In the paper, IMPALA was the chosen method

This project will focus on analyzing the C-BET algorithm and conducting simulations to understand its performance with both value-based and policy-based RL algorithms. What impacts do changes in hyperparameters have on the agent's performance?