Hatred in Scripture sometimes means to love less.
For example, Jacob loved Rachel more than Leah (Genesis 29:30). Jacob’s lesser love for Leah is defined as hatred
(Genesis 29:31). God’s hatred
for Esau was a lesser love, which was made evident by God’s sovereign rejection of him and his offspring from the underserved privilege of relationship with the Lord for which Jacob had been chosen.
1 This use of the word hate
to mean love less
appears also in Luke 14:26, where Jesus said, If anyone comes to me and does not hate his own father and mother and wife and children and brothers and sisters, yes, and even his own life, he cannot be my disciple.
Jesus means that compared to our overwhelming love for God, our lesser love for family and self appears like hatred.
However, in the case of Malachi 1:3, the hatred
of God towards Esau is contrasted with the love
of God for Jacob. God’s love for Jacob is God’s sovereign choice to establish an intimate relationship with Jacob’s descendants, in which God would seek their well being, even at cost to His Son. God’s hatred for Esau, accordingly, should be seen as God’s sovereign choice not to establish an intimate relationship with Esau’s descendants, and not to seek their well being.
3 but Esau I have hated. I have laid waste his hill country and left his heritage to jackals of the desert.”