Does the motor run OK? Does the prop angle change significantly when you wiggle it that way? Can you feel a gravelly feeling when you turn the motor, or is it smooth aside from the normal cogging? Does the spinny* part of the motor rub on the stay-put part of the motor when you're pushing and pulling it while turning it? Or is it just a side to side translation of the prop?
I suspect that a certain amount of wear-in is normal. Ball bearings in general offer you a choice between high friction with good locating, or low friction and a bit sloppy. So you actually don't want the tightest fit in a motor. "Noticeable play" is hard to quantify, though, because there's slobs out there that would only notice play in the bearings when the motor fell apart, and there's uptight detail-oriented guys who might notice a few thousands of an inch of play that no one else could perceive.
On the plus side, I'm pretty sure that changing out the bearings in these motors is fairly easy -- just get a set from Boca or Cobra, press out the old ones, and press in the new.
* I believe in using precise engineering terms at all times