"I was doing other things in life and thought show business was in my past, or in my behind as an aunt once said to me," Grant told CNN in an interview this week.
But Grant really liked the project and said he couldn't pass on the opportunity to work with Streep. Though, at first, she both thrilled and terrified him.
"Suddenly to have to be the husband of a woman that I've watched all my life be, you know, the greatest actress in the world that is...it's somewhere between a great dream and one's worst nightmare," Grant said.
"Florence Foster Jenkins" is based on the true story of a woman with little talent and big ambitions who improbably builds a singing career in the 1940s.
Grant said the jovial atmosphere on the film's set helped him conquer some serious nervousness over co-starring with Streep.
"You have nightmares, as an actor, that you're playing King Lear and you don't know any of the lines and it's a little bit close to, 'I've now got to go and do emotional scenes with Meryl Streep,'" Grant said.
Grant does some dancing in "Florence Foster Jenkins," but the 55-year-old actor said he knows his footwork may fall short.
"Throughout my life, whenever I thought I'm dancing welI, I'm not." he said. "I remember an old girl coming up to me -- I was quite young and I was really getting down thinking I was quite good -- she came to whisper something in my ear and I thought this is going to be good. All she said was 'Are you joking?' I wasn't joking! That was my best dancing."
Although Grant plays Florence's husband in the film -- and rose to fame in romantic comedies -- he said a real marriage just isn't for him.
"I'm not a great believer in marriages as an institution, or even in very long term relationships," he said. "I'm not sure we're built that way."
Now that Grant has returned to the screen in "Florence Foster Jenkins," he suggested there may be more projects in the works.
"Although I'm largely doing other things in life, it's very nice occasionally to put my toe back in the waters of show business. Even at this moment, there's something I'm sort of looking at."
"Florence Foster Jenkins" releases in theaters August 12.