I think most of the time when this happens, is is a friend trying to get you into a match counting down already. People are going in and out of games so much when you receive the invite someone else has already taken your spot.
Your friend should step out of the game and invite first, wait for you to arrive and once all of your friends have arrived then find a match. If you have a friend inviting everyone on his list, you may still not get in quick enough. Good luck.
This usually happens when your friend invites you and continues without waiting or the lobby fills. Also, this can happen when you have a bad connection or vice versa and the game gives in while loading.
This happens with every game invite i try to join, even if it is in the lobby before you click find match... i have a great connection (always full bars) and so do most of my friends... i've tried restarting my modem and ps3...