When you speak using 'to' it looks different that when speaking using other means:
You say, "Hey"
You nod at Jimmy and say, "Hey"
You [to Jimmy]: Hey.
I would propose that the output of 'to jimmy Hey.' be changed from:
You [to Jimmy]: Hey.
To:
You say to Jimmy, "Hey."
If it wasn't too hard maybe even put in some of that 'say' magic and automatically use "ask" in place of "say" when the line ends with a question mark. I'd leave everything else about the command the same. If such a change is possible, I think it would make 'to' messages blend in with others so much more nicely.