The Non-Saturating Cost Function: Changing the Rules
Early in the duel, the art forger is terrible. His paintings look like chaotic ink splatters. The inspector catches him every single time.
If the Shogun’s rules say, “You only get rewarded based on how close you are to fooling the inspector,” the forger is in trouble. When his work is utterly terrible, he is 0% close to fooling the inspector. The feedback he gets is zero. Without a hint of how to improve, he gets demoralized and stops learning. He is stuck at the bottom.
To fix this, the game master changes the rules. The new rule says: “Instead of worrying about how close you are to winning, focus entirely on how often the inspector fails.”
When the forger is terrible, the inspector never fails. But by shifting the focus, the forger’s earliest, microscopic improvements suddenly yield a huge learning signal. When his drawings are at their worst, the new rules ensure he receives the strongest feedback on exactly what went wrong, giving him the massive push he needs to improve.
The Reality
In AI, this represents a crucial mathematical fix called the “Non-Saturating Cost Function” (or Non-Saturating Loss).
When we train a GAN with the standard Minimax game, the Generator suffers from “vanishing gradients” at the beginning of training. If the Discriminator is much better than the Generator (which is true at the start, since the Generator only outputs random noise), the Generator’s loss function goes flat (it “saturates”). The AI learns absolutely nothing.
To fix this, we change the mathematical goal. Instead of telling the Generator to minimize the probability that the Discriminator is correct, we tell the Generator to maximize the probability that the Discriminator is wrong.
The Why
Mathematically, these two goals seem identical. But in the landscape of calculus and gradient descent, they look completely different. By making this simple swap in the equation, the Generator gets a massive learning signal (steep gradients) precisely when its data is at its worst. This is the mathematical trick that gets GANs off the ground and stops early training from failing completely.
The Takeaway
When someone is failing completely, they don’t need a zero score; they need the strongest, most detailed feedback on how to improve.
AI specialists call it: Non-Saturating Cost Function (Log D trick)
To prevent vanishing gradients early in training when the Discriminator dominates, the Generator’s objective is modified. Instead of minimizing log(1 - D(G(z))), it maximizes log(D(G(z))), ensuring strong gradients and significant parameter updates when the generated samples are poor.
💬 Have you ever tried to learn a complex new skill, but quit early because the initial feedback loop was too punishing?
Part 8 of 14 | #GenerativeModelsForHumans
#ai_edu Based on Stanford and industry lectures