An affair isn't the worst thing that can happen in a relationship.

By: Bethany Allendale and Stu McLauren for The Good Men Project.

Pretty much everyone who is in a committed relationship agrees that if they found out their partner had sex, even once, with someone else it would be a challenge to move past it and stay together. That’s the big one-event shake up that most people assume is the beginning of the end if not the marker for the end itself.

While both of us would be nonplussed if the other had an affair or one-night-stand, there are three patterns that we’ve even fallen prey to that we know could be (almost) silent killers for us and probably any other couple.

1. Withholding attention.

You’ve heard about “ghosting” right? Pretty extreme and most people wouldn’t dream of doing it to the love of their life, and most of us would notice pretty quickly if we were being ghosted.

But there are so many ways of withholding attention that aren’t even intentional. Preoccupation – you know, like with your phone, or the television (we don’t have one of those, but the internet is a close runner up.) Or maybe just preoccupation with your own thoughts – we’re both prone to getting lost in our heads, in the clouds, or in time. Or shutting down – crawling into the shell of self to process, to stew, to ponder, or just to hang out and relax.

Even if it’s not intended to hurt, when you withhold your attention from your partner, your lover, that person who is in your life because they get off on being with you, you’re committing a “micro-aggression” toward that thing that keeps you together.

And if you are withholding attention as a way to prove a point, or strike back, stop that right now. Everyone needs to withdraw and unplug but don’t retreat in order to get even. Hurting someone emotionally always leaves a mark.

2. Withholding trust.

Everyone brings past relationships into the current dynamic. Whether those relationships were with spouses, lovers, friends, siblings, parents, authority figures, or the Grinch next door, they create patterns of trust, and lack of it.

None of those patterns are about your partner, but they certainly affect your partner. We’ve navigated land mines and dungeons of trigger wires and green-eyed monsters until we can guess when something is a “this is about me not you” kind of thing, but when either of us senses a lack of trust it can still range all the way from a minor road bump to a steep hill.

But if you’re withholding your trust for a reason, justified or not, a shifting of boundaries or treating a partner with suspicion is always going to result in tension. Deciding to withhold trust is one of the deepest betrayals you can inflict on a partner. It puts up a wall, and reflects their worst traits back at them. No relationship, however loving, can survive that for very long.

3. Withholding intimacy.

Intimacy isn’t going through the motions. And it isn’t (always) a romantic performance or expectation. Intimacy is meeting on the same, shared plane of experience. On purpose.

Physical intimacy is important to a romantic partnership. Actually, for us anyway, it’s vital. It’s not just about sex, that’s only one of many ways we physically demonstrate our love for each other. But there is nothing wrong with saying that sexual intimacy is a cornerstone of our physical intimacy. But emotional intimacy is even more important. Without that the physical or sexual intimacy is just “off.” We’re not really in it together if there is an emotional barrier between us.

It’s perfectly natural that there are times one or both of us will just need to be left alone. For one thing, we’re both introverts (highly social, but definitely introverts.) For another thing, we’re human. So it’s going to happen.

WATCH Michele Weiner-Davis' TED talk on sex-starved marriages. Post continues after video...

But if either of us ever withheld intimacy — of any kind — not from our own need for personal time and space, but as a tactic or punishment, or if we ever displaced that intimacy usually reserved for each other onto another person, it’s likely that we would cause wounds that would take a long time to heal – if they ever did. We both have intimate friends, but those friends never come first for either us. First is always reserved just for each other.

None of these patterns are the instant end game that a sexual fling might be, but they are all slow, steady drips of betrayal that eventually erode even the strongest relationship.

This post was originally published on The Good Men Project.