Job 31:28 (ESV)

28 this also would be an iniquity to be punished by the judges, for I would have been false to God above.

Job views idolatry as a sin to be judged, just as adultery (see Job 31:11). Unfaithfulness to God was a sin that he would not tolerate, neither in himself nor in his children (Job 1:5).