That's weird i have the opposite problem, i can only connect from the story mode (single player), and only of someone invites me. Luckily I have two xboxes and two copies and so my husband can invite me because for some weird reason he has no problem getting in at all. I never get kicked when we are in a party tho.
I have seen the invisible players as well (haha not really seen per say...), and wow, what a bummer on those glitches, nothing worse than getting your ass handed to you over and over by a cocky invisible 12 year old.