"I don't know what you're saying, chum."

Real simple. If you directly talk to an NPC who has talk responses in a language they don't respond, have them tell you that they don't speak whatever language you're using.