But most old people seem to be irritating.

Only to an irritable person. What you are talking about now is a reaction to something. Only an irritable person can get irritated; this is something that comes from within, not from without. Of course, irritability will react with irritability and perhaps this is what you mean.