I would never do this, but I have to admit that it sounds like an effective tactic. Unless, of course, there’s no way to easily excuse yourself and you are forced to pretend to have a conversation with the fake “caller” right in front of the person you’re trying to avoid talking to.
I know I wouldn’t be able to keep it credible. I’d either sound too vague or else, even worse, I’d try to come up with some specific topic and make up a conversation so outlandish it would give me away. It would be like a bad Bob Newhart routine. I would try to think of some subject to fake talking about and I’d end up saying:
“Hello, yes? Oh, hi! Oh, really? Well, I’m sorry to hear you were abducted by aliens from the Andromeda galaxy…”
People would get suspicious.