Say "No" to "Yeah"

Some of the cultures consider saying "Yeah" as a disrespectful word. Yesterday I was talking to an elderly person. A part of Conversation went as,

Himself : Kesava Krishnan?
Me     : Yeah!
Himself : You mailed me?
Me     : Yeah!

Himself : Say "Yes" or "No" for my questions, don't say "Yeah" because I don't like it.

At that moment only I understood that using "Yeah" is disrespectful! I wish to say Sorry to that person to whom I talked in disrespectful manner. I never did that intentionally or to show my proudishness. :( Anyway advice to all is not to use "Yeah/Yup/Nah/Nope/gonna" and like words with elders.