The short-short answer, and until you answer Brent's questions, all I can give: open the needle, maybe longer lines, and trim, trim, trim.
Basically, speed covers a wide swath of trimming sins, so if you can't just open the needle a bit and have the plane fly nice, then you need to change the plane somehow. If you're lucky you just need to trim it; if you're not lucky you need to repair warps, or abandon it and start over.
You should be able to get lap times to 5 or 5.2 seconds with a well-built plane and fair trimming abilities. If your plane is a twisted-up horror or if you built it around a brick then you won't be able to do even that.
Follow
this link, and read all the articles on trim, starting with chapter 1. If you want us to give you guided advise, rather than just bloviating randomly, answer Brent's questions.
If I were to start a School of Stunt, I wouldn't let anyone fly a flapped stunter in competition until they were earning around 350 points per flight (I wonder what that would be in FAI rules?). Flapless stunters are much easier to trim -- you take out at least half of the variables along with the flaps -- and having flaps on a plane isn't going to change your standings by all that much when you're down at that level. It's much better to be able to concentrate on flying rather than on trimming and repair.
And: however fast the plane seems to be flying
now, in a year, even if you don't slow it down, it'll seem to be flying much slower. There comes a point if you practice hard enough where things start to click, and you just find yourself ahead of the plane instead of behind it. At that point "improvement" becomes a matter of making shapes better and getting your bottoms where they belong, instead of a matter of making it through the pattern without a crash.