I have found that, for some reason, it is easy to cause the allen wrench (especially cheap, soft wrenches) to slip inside the set screw, thereby deforming the socket just enough that the proper allen wrench just won't bite enough to loosen the really tight set screws.
In most of those cases, I have also found that if I have a Torx wrench of the right size, I can drive the Torx into the set screw socket and get the little thing right out.
Cheers,
Jim