I invited the man who I had an affair with to my wedding, should I reconsider?

I had an an affair with a married man and have secretly kept a relationship with him while dating my fiance. I invited him to my wedding. I never thought he would come. He replied that he and his wife will be there. Is this too shameful to have on my wedding day? Should I call him up and tell him not to come? My fiance thinks he is just an old friend and his wife has no idea.