It probably wouldn’t be as big a deal if it were a platonic relationship. The reason the sex matters is that sex tends to lead to people making bad decisions. Also, the fact that it’s an affair matters because lying and sex in secret also is associated with bad decisions.

If he were in a platonic relationship with her but they were caught doing heroin together, that might also be a big deal because drugs also lead to bad decisions. Illegal drugs done in secret would be an even bigger deal.