Questions: Can a venturi be too big that fuel just can't keep up?
Absolutely - but apparently not in this case, since it is keeping up just fine at lower RPM. A smaller venturi might help because it reshapes the HP curve, rolling off the power at a lower RPM, which is the essence of your problem - you are running below the peak of the HP curve, which is unstable. As soon as it speeds up, it has more power at the higher RPM, so, it stays there. There may also be excessive drag in the fuel system due to a excessively small restriction/hole somewhere in the fuel system.
Does higher castor fuel keep an engine from going lean?
Generally not, if anything it makes it less able to draw fuel.
Are some engines just lousy?
Yes.
Does underpropping make an engine go lean?
Define "underpropping", but if you mean, too small, then no, too much prop will reduce the fuel draw and cause it to go leaner, and particular, too much pitch will induce you try to run it too slowly (to get a reasonable lap time), since you are on the unstable side of the RPM curve, the first chance it gets, it will take off to a higher RPM. A potential solution is to run less pitch, which will require more RPM, which can put you on the stable side of the HP curve. If it is too fast, you can also reduce the diameter to control the airspeed and require even more RPM. Thats how the 20FP works.
Without looking at your problem and doing some experiments, I hesitate to tell you exactly what to change to get it to work, but to forestall the ministrations of the Head Gasket Patrol, it is extremely unlikely to be that the engine is "overcompressed" and adding head gaskets is more likely to exacerbate the problem than to fix it.
My suggestions are:
Inspect the fuel system and in particular the inlet side of the spraybar for fuel passages that look too small or out of proportion to the fuel tubing, or any kinks or pinch points that might restrict the fuel flow.
Try conventional fuel, like Powermaster "Air" 10% or SIG Champion that has much less castor oil and therefore less viscosity that GMA fuel, same goal, to reduce the fuel flow restriction
Measure the venturi and spraybar dimensions and calculate the venturi choke area, report back, but if it is >0.024 square inches its probably too big, particularly if it has a fuel outlet that is flush with the inner wall of the venturi (i.e. just a hole in the side where the fuel comes out).
Run less pitch, like a 12-4.
Brett