Hello Heman,
Governor High mode is probably what you want. Simply setting a timer up by taking a stab at the pulse width and then making large adjustments will make you frustrated.
I do recommend that you buy a servo pulse generator/meter that will allow you to control the motor and ESC and actually see what control pulse width you need to achieve the RPM you want to start with.
Then you set up the timer and measure its output with the same meter. Adjust until it's right, and then (in my opinion) you are finally ready to put the beast in the air.
For one thing, you will find that the relationship of RPM to control pulse width is not linear. Small changes in pulse width (equivalent to only two or three percent on the timer adjustment) may change the RPM by 400 or more!
Some timers have better or finer resolution steps than others. All this requires a pulse generator/meter. They used to cost a bit, like the VEXA RC unit I still use but now the import places sell them for less than the plastic case would cost, here at home. Go figure ...
I don't recommend running in SET RPM Mode (though many pilots are completely happy with it!) because I like to keep the option open to tweak the RPM up 100 or 200 RPM for changes in weather from morning to afternoon heat. This requires a timer that is easy to set up. As moderator I try not make a recommendation, but if you spend some time surfing through older threads and reading the "list your setup" section you will see what works for flyers ranging from Sport to high level competitors.
As far as your jumping RPM problem is concerned, try this for some bench tests: put a small prop on the motor just so you can tach it. Then take your timer and set it for 30 seconds run and make a chart of RPM versus pulse width percentage. Try 10%, 20%, 30%, 40%, ... I'll bet that all the RPM from 40% on up are the same in RPM High Gov. You'll probably end up running close to 30%, and 1% steps will be several hundred RPM. When you ask for 80%, you are probably asking for an RPM that the motor/prop/battery cannot make and the ESC produces unpredictable garbage.
Got an RC flying buddy who will lend a TX and RX to the cause? Maybe that will allow you to ballpark the percentage quickly by looking at the TX stick movement.
I hope this helps ... meanwhile go searching through the listed setups for something similar.
take care,
Dean P.