"Why can't we be more in sync?" many women wonder. "Why don't we want to have sex as much as our husbands do?"
"Mismatched libidos are the norm," said Holland, a psychiatrist who has practiced in New York for 20 years.
And it's not always the men who are wanting it more. Some women desire sex more frequently than their male partners. And same-sex couples grapple with the issue, too.
Holland says since our sex drives are rarely 100% in sync in relationships, we need to be honest with ourselves and our partners about what we want when it comes to sex and when we want it.
"Sometimes you want a gourmet dinner and you want roasted chicken and a potato galette and at other times, you're OK with nuggets and fries and you're in a hurry," she said. "You have to be able to sort of communicate where you are in your (menstrual) cycle because your libido is very much tied into fertility."
She continued, "If you are not on the pill and you're a free-range cycling woman, you're going to be more libidinous when you're fertile."
'Men warm up much more quickly than women'
Transition time, many women say, is also crucial. I've had friends joke that they simply can't go immediately from emptying the dishwasher to sexual encounter, while the same shift is no problem for their husbands.
"Men warm up much more quickly than women do," said Holland, noting research that shows that a man can orgasm in about four minutes while it may take a woman 20 to 30 minutes to climax during sex.
"So when a man and a woman get together to have sex and the man can easily climax in four or five minutes but the woman takes a half hour, what do you do with that discrepancy?"
Yazel of Cosmopolitian said her magazine's recent orgasm survey
of women 18 to 40 showed that time to get in the mood is critical for many women.
"What we found was that the reason that women couldn't have orgasms most of the time was that they just reported they just couldn't get over the edge, and to me that says you didn't have a transition. You weren't quite in the mood and so I think that's the real issue, is sort of making sure that you're ready to go there."
Hostin, who's also a mother of two, said yes, women need more transition time, but her belief is that if your husband wants to have sex, then you have sex with your husband. You don't turn him down.
"I think you do it and it's something that I tell my girlfriends because ... we also don't need him to go somewhere else," she said.
"I think that in a marriage you do a lot of things that you may not be in the mood for. Am I really in the mood to cook tonight? Am I really in the mood to listen to your story about work? ... No, and so even if I'm not in the mood, I think that as a friend, as a lover, as a partner, you make sure you get in the mood."
In her book "Moody Bitches," Holland jokes that sometimes you just have to have sex with your husband "so he'll get off your back and you can sleep, like literally."
Thinking that you love your husband and you are going to do this for him and "it won't be terrible" is OK, she said.
Resentment: A 'huge libido killer'
Mood is key, though, said Yazel, who is also a mom of a 4-year-old. It's one thing to not be in the mood and another to not be feeling any desire whatsoever. "No one should do this if they feel bullied," she said.
Resentment is a "huge libido killer," said Holland, a mom of two who is also the author of the best-selling memoir "Weekends at Bellevue."
It is "the flip side of accommodation," she said. You go from, '" 'OK I can do it, I can do it,' and then all of sudden, you're like, 'No, I'm not going to do this.' "
What I've heard from many women is that sex feels like another item on the lengthy "to do" list: need to take care of the kids, finish up work assignments, get ready for the morning routine, and, oh yes, also have sex with the husband.
"That's part of where the resentment comes," said Holland. " 'Oh, I have to do this for you and you're on my to do list,' but the truth is it's also for us."
There are plenty of "feel good neurochemicals" that will "start to bubble up if you have sex and it will help your mood," she said. "And orgasms absolutely help your mood and they help relax you and they can help you get to sleep."
So what's a woman living in a relationship with mismatched libidos to do?
Besides speaking honestly and openly with your partner and being honest with yourself about what you want and how you really feel, Holland also recommends something else.
"I always tell my patients go ahead and just start to have sex because what you may find, which is really true, is that once you're cuddling and touching, it gets oxytocin going, you get pheromones. You get testosterone. You start to get horny. If you actually just kind of dive in and go ahead and start kissing and cuddling and caressing, you may discover that you actually are in the mood after all."
What do you think is the best way to deal with mismatched libidos? Share your thoughts with Kelly Wallace on Twitter @kellywallacetv or CNN Parents on Facebook