The retired American swimmer, who won 23 Olympic gold medals, urged greater international policing of doping in sport.
He told a US congressional hearing on improving anti-doping measures "the time to act is now."
"I don't believe I've stood up at an international competition and the rest of the field has been clean," the 31-year-old said Tuesday. "I don't think I've ever felt that.
"I know that when I do stand up in the US, I know we're all clean because we're going through the same thing.
"We're going through the whereabouts, the out-of-competition test. We're doing all of that stuff.
"Internationally, I think there has to be something done, and it has to be done now."
'Uncertainty and suspicion'
The Rio 2016 Olympics were held in the shadow of the explosive McLaren Report
which detailed Russia's state-sponsored doping program.
Russia has always denied government officials were complicit in doping the country's athletes.
"There has never been any institutional conspiracy to conceal positive doping tests in Russia and there never will be, on the contrary, doping will only be fought against," President Vladimir Putin said Wednesday in response to calls from the association of National Anti-Doping Organisations (NADOs) for a temporary blanket ban on Russia from international sport.
Phelps said that prior to and during last summer's Olympics there was "uncertainty and suspicion" which undermined the "belief and confidence" he and his teammates had.
In a 1,300-word testimony, published ahead of his appearance in the House of Representatives, Phelps wrote: "Throughout my career, I have thought that some athletes were cheating and in some cases those suspicions were confirmed.
"Given all the testing I and others have been through, I have a hard time understanding this.
"I can't adequately describe how frustrating it is to see another athlete break through performance barriers in unrealistic timeframes, knowing what I had to go through to do it.
"I watched how this affected my teammates too. Even the suspicion of doping is disillusioning for clean athletes."
Phelps said he was tested "almost every day" at Rio 2016, but questioned: "Are there people going through the same as I do?"
Phelps was one of five witnesses called to testify before the House Energy and Commerce Subcommittee on Oversight and Investigations.
He was joined by Adam Nelson, 2004 Olympic shot put champion, US Anti-Doping Agency chief executive Travis Tygart, Dr Richard Budgett -- the International Olympic Committee's medical and scientific director -- and Rob Koehler, World Anti-Doping Agency deputy director general.