Thanks for noticing this issue. I've kept repeating this mistake since I read "Human-level control through deep reinforcement learning" in the wrong way.

Here's a code snippet from the DeepMind implementation of dqn (NeuralQLearner.lua) available at https://sites.google.com/a/deepmind.com/dqn/

if self.clip_delta then
    delta[delta:ge(self.clip_delta)] = self.clip_delta
    delta[delta:le(-self.clip_delta)] = -self.clip_delta
end

Doesn't this mean that the original implementation has the same issue? Maybe this should be mentioned as a difference to the original implementation?

@vuoristo The original implementation directly clips the error term for the gradient, as described in the paper. delta there has the same name and value, but serves different role. It essentially goes to the second parameter of backward(), and the function receives gradients with respect to the output of the module as a second parameter. I am pretty sure that both implementations clip the same thing now.

Hi, I believe that the linear range of the Huber function should be 2. That results into:

def clipped_error(x):
    return tf.where(tf.abs(x) < 2.0, 0.5 * tf.square(x), 2*(tf.abs(x) - 1))

Explanation
I believe that Huber loss was primarily used to clip the gradient resulting from Q value overestimation. E.g. if your target Q_ value really differs from the current Q value in:

Q(s,a) -> r + max Q_(s_, a)

You don't want to do large updates, because this is an estimation anyway. This could also be the only source of large updates, because you clip the reward to [-1, 1] anyway.

But to qualify as correct Q-learning, you want to compute an estimate over possible future states s_. This what you would get with MSE, or Huber loss with behaving as MSE in [-2, 2] region. 2 is the distance between the possible rewards.

Intuition:
If your current estimation for a state s and action a terminating the episode is Q(s, a) = 1 and now you get a reward of -1 (in stochastic environment), the loss is L = huber( 1 + 1 ). And that's where you want it to behave as MSE.

Note: Some additional learning rate tuning might be necessary.

Discussion welcome. Opinion from @karpathy welcome.

reward is clipped, but not the Q value. Distance between Q values can be larger than 2.

@ppwwyyxx I assume you say that the error of Q can be larger than 2. That's true, and that's where Huber loss helps - avoids large updates. When it converges, the error is 0, however.

What I was saying is that because the maximal distance between possible reward is 2, the delta argument of Huber loss function should also be 2. With this loss function the Q-learning should converge to the same solution as with MSE - giving proper Q value estimations (means):

Q(s, a) = E[ r + max Q(s_, a) ] 

That's how the Q-learning is defined.

With delta=1, it converges to a different solution, where Q values are different.

Q(s, a) ≠ E[ r + max Q(s_, a) ] 

Also see corresponding reddit discussion.