I guarantee that if you "throw it all out the window and play what you feel from the heart" it's going to sound like shit.
A good instrumentalist knows what notes to play during improvisation because they have a general understanding of how to stay in key and make specific choices when playing accidentals. They know this because they've played and practiced for a long time; they may not be theory experts but they still know what's going to work and what isn't going to work. An instrument is a tool used to manipulate sound, and instrumentalists have to achieve a certain proficiency with that tool in order to produce the sounds they want. That means actually knowing how to play an F# or a Bb when they need to, and that means they're making the choice to play an F# or a Bb for specific reasons. That choice can be made within an instant, but it's still a choice that's being made.
I always roll my eyes a little when people talk about improvisation in these really romantic, cerebral terms where a person is just "feeling the music." There's a lot of study and practice that needs to be done to get to the point where an instrumentalist is proficient enough to be able to make those choices sub-consciously and let muscle memory guide their playing.