depends, I start with what I need to use to get rid of the testure I am getting rid of,, it takes what it takes,, if your primer is nice and smooth with no orange peal, then 320 would be fine, if you have orange peal, ( or grapefruit skin)

sometimes its better to hit it a good one shot with 240 to cut the tops off,, My avenger was finish sanded with 320,, no lie,, that was my final sanding on the surfacer,, because I knew I was spraying a sealer coat over it and that would hide the 320 scratches,,
OBTW, next column,, sandpaper