Back when I started watching television, it would take me ten to twenty minutes of half-understanding before I'd find myself mostly following along without too much need for subtitles. I don't know if it was just getting familiar with the words and accents, but I think there was something neurological there.

It was the same for radio incidentally. These days I find it much easier to just start listening, which I guess is a sign of progress. I used to get really tired after watching Chinese television too -- be curious if other people have had this experience too, since I've wondered if it was related to learning.