We can assume that these are the same people mentioned in Ezra 3:3. They are the people living in the areas immediately around Jerusalem: Samaria, Ammon, Ashdod, Edom, and Moab.1 They are identified as adversaries because their desire is to see the people of Israel come to harm, which will be explicitly shown later in the chapter.
1 Now when the adversaries of Judah and Benjamin heard that the returned exiles were building a temple to the LORD, the God of Israel,