If you wish to divorce him you must ask yourself what doing so will accomplish spiritually. Will divorcing him glorify God? Will he see Christ in you? Will you grow and change into the image and likeness of Christ through divorcing him?
If we take these questions one at a time perhaps we can glean some wisdom.
Will divorcing him glorify God? Scripture says God hates divorce (Mal. 2:16). A divorce is a breaking of a covenant that two people make before God and witnesses that they will become one flesh (Gen 2:24).
When he committed adultery he broke the fidelity part of your covenant. This does not glorify God.
Does his breaking his part of the covenant necessitate you break yours? Hardly. You can choose to remain faithful to your covenant.
What glorifies God is forgiveness, mercy, and grace extended to those who least deserve it. Your husband sinned against God, you, and the woman he was involved with. He does not deserve your forgiveness, grace, or mercy. However, you must ask yourself if you deserve forgiveness, grace and mercy from a holy God! The answer to that question is no, you do not.
Therefore, as the elect of God, holy and beloved, put on tender mercies, kindness, humility, meekness, longsuffering; bearing with one another, and forgiving one another, if anyone has a complaint against another; even as Christ forgave you, so you also must do. Colossians 3:12-13 (NKJV)
God's Word leaves us no option. When we are wronged we must forgive. Does this mean you trust him again? No, not right away! He may not even be willing to cease the affair or cut off the relationship. He has to earn trust but grace and mercy are given to him freely because of Christ.
Then Peter came and said to Him, “Lord, how often shall my brother sin against me and I forgive him? Up to seven times?” Jesus said to him, “I do not say to you, up to seven times, but up to seventy times seven. Matthew 18:21-22 (NASB)
Your husband is your closest brother on earth...
I have said before that I do not believe we are to verbally grant forgiveness to someone who has not asked us for it. In reality, the verbal granting of forgiveness is the last step of the process. Forgiveness begins in the heart and that is where that important transaction takes place between you and God. Forgiving won't be easy, especially if he is a repeat offender, but it is possible because of Christ and the supernatual power of the Holy Spirit. Desiring to forgive him because it glorifies God is the first step. Understanding that forgiveness brings God glory and honor is much more important than how you or I feel about something. What a blessing to be able to look your husband in the eye if and when he asks for your forgiveness and say, "Honey, I already have forgiven you."
What a wonderful representation of the Lord Jesus Christ you will be should you choose to be forgiving and honor God!