Nagarjuna said:
Although Buddhas call objects adversaries, it is not people,
Rather, it is the delusions in their minds that are the real foe.
If someone hurts us, it is because of the delusions in their mind.  Therefore Buddha said that we should not consider the person as an enemy, but have compassion for them and consider the delusions in their mind as the actual enemy.

Written by Khenpo Karten Rinpoche in the book Precious Teachings, the chapter entitled “Happiness and Suffering Are Only Appearances, They Are Not Real” page 102 in the paper version and 108 of the PDF version. 

Image from Wikimedia, artist unknown.