Years ago when this "problem" of "DRONES" and first person flight stuff started getting news, I too immediately thought how easy it would be to just shoot them down if over my property.
Then I read and learned more about the subject: There are many legitimate uses and for these craft, And for a lot of us rural folks, the likelihood of seeing a camera bearing quad over or near our property is pretty high. And the notion that it might be some snooping sheriff, police, or voyeur is actually fairly low.
In my area there are several crafty long time RC pilots who have created niche companies to provide close overhead video surveillance and photography; Weddings, lost child searches, lost animal (mostly cattle) searches, power line inspection, property survey (county), property survey (real-estate sales), Fish and wildlife surveys (how many deer and what sex are accumulating in and area etc). I am sure there are other useful ways these craft can make the operator money for providing a service
The obvious potential for Fed and local law enforcement to snoop is a worry for me as I distrust my government these days....that said, I suspect IF the Government was trying to see if I am growing pot they probably would have some worthless (to me) warrant and IF I shot down their surveillance gear I certainty would NOW be subject to more grief than I ever want from the "LAW"
Since we are 60ish and no longer run around the backyard naked, and I do not cook meth or grow pot, I think IF I ever noticed one hovering overhead, I might just be curious what it is doing and then go back about my life of enjoying my wife, children, grandchildren, pets, friends, and hobbies.