I have a new "Top Hat" that is unfinished for a few years only due to priorities. It is covered with polyspan, and yes I did the same thing, Minwax and all.
Without using names, a good friend who is one of the best finishers that I know, said to shoot autoprimer and the fuzz can be made to disappear. I still have to try it so I can't tell how much to sand off, but I would guess that just like other filling, you might have to prime some areas a second time. Done correctly it shouldn't add any weight.
This sounds feasible, but I haven't tried it yet.
Oh BTW, Polyspan has a shiny side that is the smooth side to be up. This is also the side on the outside of the roll.