Two possibilities that I can think of:
It's staying in 5.1 mode when you switch to TV, which is probably only in stereo. You will lose much of the sound because there's no center channel for voice.
Or it's simply changing itself to stereo mode when you switch to the TV, and getting quieter because there are only two speakers going instead of five.