Lot's of good suggestions already.
Collets are funny and hard to deal with. A tip I use when tightening a collet is to shove my finger into the motor compartment and wedge it between the fuselage side and the outer case of the motor (on an outrunner). Just turn the motor until your finger gets wedged in there a bit. This will hold the outer case of the motor while you tighten the collet. In my experience, it really does not take much pressure to tighten the collet. Your wedged finger will move as you tighten, so once you have snugged the collet up to the shaft, it really can't go much further before it is plenty tight - it's already touching the motor shaft. Do too much and you WILL over tighten the collet and then you are hosed (ask me how I know
.
In your case, the outer case of the motor is right there, just grab a hold of it with your thumb and finger and tighten the spinner and collet until you can't pull the collet and prop off. This is plenty tight. If they still come flying off when you run the motor, you have other issues. Better yet, take the motor out and try this same technique. Holding the outer case, slowly tighten the prop and collet until you cannot pull them off the shaft. This will show you how little tightening is required once the collet is snug to the shaft. If you can still pull the prop and collet off, make sure your prop is not keeping the collet from properly tightening as Dennis mentioned. Collets are made to "bottom out" - in other words, the collet should extend as far as possible onto the shaft. A small amount of "not bottoming out" (maybe for needed prop clearance), generally won't hurt you, but if you are not close enough to bottoming out, you can set up a wicked vibration.
The second item I would check are the mounting screws coming through your firewall. While your motor is out, measure very exactly the thickness of your mounting firewall (in case you were a bit overzealous sanding it). Carefully run the mounting screws into the motor case to see how far they extend into the case. Deduct for the thickness of your firewall. If the screws are too long, they can hit/rub the armature and lot's of bad things can happen from there. Could have a rub which induces a lot of vibration. Vibration is not your friend
The screws can be flush on the inside of the motor case, but you really don't want them extending into the armature area.
Your motor appears to be a 400 size of the Chinese variety, so it would not be totally unheard of to just have a lemon. I order some of these directly from China for our "First Flight Training" aircraft. I get the motor, mount and speed control for $6.38. Try another motor and collet if nothing else works.
If you have already over tightened the collet the first time you put it on, it could be hosed - don't be afraid to get another collet.
Sounds like you made sure the motor and prop are not rubbing the aircraft, but do as others have said, double check for possible foreign parts and make sure none of your wiring is touching the motor once it starts up.
We have faith - you'll get it figured out.