📘 Explanation: The Case of the Infinite Return

Your analysis of the evidence is correct. The Return just keeps getting more negative, heading towards negative infinity. This presents a fundamental problem with our current definition of "Return". Before we reveal the solution, take a moment to think like a detective: - **What is the root cause?** What fundamental property of this "black box" environment is causing the return to grow infinitely? - **How could we fix it?** How could we change our calculation of "Return" so that it results in a stable, finite number, even if the simulation runs forever?

Click here to reveal the official explanation

The property you've uncovered is that our environment is a **Continuing Task**. - An **Episodic Task** is one that has special terminal states (like a goal or a pit) that end the simulation. - A **Continuing Task**, like our current one, never ends on its own. In a continuing task, the simple sum of rewards is not a useful metric because it almost always diverges to infinity. The standard solution is to introduce a **discount factor**, called **gamma (γ)**. **The Solution: Discounting** Gamma is a number between 0 and 1 that represents the agent's "patience." It makes rewards that are far in the future less valuable than rewards that are received immediately. Instead of the simple Return, we now calculate the **Discounted Return**, where the reward at step $k$ is multiplied by $\gamma^k$: $G_0 = R_1 + \gamma R_2 + \gamma^2 R_3 + \gamma^3 R_4 + \dots$ When $\gamma < 1$, this sum is guaranteed to be a finite number, even if the sequence of rewards goes on forever! This gives us a stable metric we can actually use. Let's put it to the test.