A lot of the Vito storyline hate comes from it being introduced in the last group of episodes and people not expecting that kind of drawn-out storyline with a minor character so late in the series. And I'm not saying it happened in these parts, but the fact that it was a gay storyline turned off a lot of the neanderthal types that watch the show, which to me was worth the price of admission right there.

I also really don't get why people say it was "filler". Filler is something you do that's safe and bland to pad things, but there was nothing safe about it. They knew it was going to be controversial. Whether you think he completely succeeded or not, introducing a gay storyline into a mafia show was probably one of the bravest things Chase ever attempted.
Plus, when it's all said and done, I think the series as a whole will be stronger for attempting something different like the Vito storyline, as opposed to introducing something the series had already done before, like a new baddy for Tony to butt heads with.