I had one like that. I made the mistake of using white glue on an imperfect saddle. It started straight but the glue pulled it crooked.
That is always an issue. White glue is particularly bad about it - it both wets the wood, causing it to swell, and the glue itself shrinks, with unpredictable results. It has lots of water compared to something like Titebond. Ambroid and other model cements are the dead worst, of course.
My usual approach is to get it aligned, then tack-glue with thin Hot Stuff. Let it sit for a while or overnight, then remeasure it, if no good, crack it loose and scoot it whichever way it goes, try again. Work up to it, once you are good, tack it better, wait an hour, measure again, repeat, until it is very solid in all the likely trouble spots and you can't easily move it. Then glue it solid with more Hot Stuff.
Letting it sit between cycles is very important, because you usually have to nudge/slide/push things around to get it to line up. You need to give it time to relax and attain a stable new position, then see if it stays. If you hard-glue it all at once, you have to do something else more drastic (like Kevin's case). I know what other people do for alignment, and someone like Paul would be horrified at my lack of alignment jigs, but I have had very repeatable results, as hacky as I am.
Brett