All of your points are perfectly valid, Tensei. When there aren't that many people on the map, the spy is at both an advantage and disadvantage: It's easier to maneuever yourself behind the team, but it's harder to blend in. If there is no engineer to deal with, the viable targets are only the demoman, soldiers, and possibly medics (since they're faster than spies they're harder to get, and any decent medic knows to not just face forward). Granted, spies could perform reconissance pretty well (especially with the cloak and dagger, but updates are generally banned from scrims and such), but it takes away a player from the team.
Personally, I'd still like to try out playing spy on a team of well-coordinated players: Deadliness increases substantially if a spy works in tandem with his teammates. I would not, however, be adverse to playing medic or soldier either (pretty good at medic, okay at soldier).