I bet that was exciting for the pilot! Of coarse with no ground referance I guess it would be less exciting.
Yeager got it completely unstable at about mach 2.4, knocked himself out on the cockpit, but came to in time to recover and land safely. They limited it to around mach 2 max for the rest of the program. That was the bane of all the early supersonic airplanes - lack of directional stability at speed. Something similar happened on the X-2 , but at mach 3.2. and that ended up being fatal.
The X-15 was perfectly stable in its natural state all the way up to mach 6+, but one of them did go unstable from a control system issue and went into an inverted flat spin on re-entry at about mach 5. Somehow the pilot broke the spin but it went unstable for another control system problem and eventually broke up and of course killed the pilot. The problem with mach 6+ for the X-15 is that it darn near burned up despite being coated with ablative heat shield material, permanently damaging the underlying airframe, and couldn't safely be flown again. That's the one in the Air Force museum and still holds the speed record for "conventional" aircraft.
I am not sure what you mean about no ground reference, both the original XS-1s and the later A, B, D, E all had decent outside visibility.
Brett