You can let it slide with a "oh?" or you can take it seriously.

Of course he thought it was a joke and let it slide with a "hm?" but he was seriously annoyed. I mean, just don't say things like that even as a joke!