(CNN) -- Richard Friedman lost more than $3 million to Bernard Madoff's investment scheme. Following the disgraced broker's guilty plea Thursday, Friedman said he hopes Madoff lives a long time.
Richard Friedman says Thursday the IRS should not keep income taxes that investors erroneously paid.
"He is going to be sentenced for 150 years. I hope he lives a very long life," said Friedman, one of several victims who was allowed to sit in the courtroom for Madoff's plea.
Madoff, 70, pleaded guilty to charges that he swindled investors through his firm, Bernard L. Madoff Investment Securities LLC. Investigators said he didn't invest any of his clients' funds into securities for more than 13 years.
A judge accepted his plea on all 11 counts, and Madoff faces up to 150 years in prison. He is scheduled to be sentenced June 16. Anatomy of the Ponzi scheme »
Friedman said Madoff isn't the only one who received an undeserved windfall via a Ponzi scheme that is estimated to have bilked thousands of investors out of a collective $65 billion.
"Madoff investors paid in hundreds of millions, if not billions, of dollars in income taxes on phantom income that never existed," he said. "The [Internal Revenue Service] should not be allowed to keep that money." Watch Friedman take the IRS and SEC to task »
Bennett Goldworth said he lost 97 percent of his net worth, $4 million, to Madoff's boondoggle. He was retired and living in Florida when he learned of the scam, which forced him to move back to New York to live with his father.
He, too, has qualms with the government, he said.
"They've done nothing in this whole process to address the victims. We've been stripped of everything," Goldworth told CNN before entering the courtroom. Watch Goldworth talk about getting closure »
New Jersey state Sen. Loretta Weinberg said some victims don't even know how to file their taxes, and the IRS has offered no guidance.
Weinberg, who lost money through indirect investments to Madoff's scheme, said she is almost angrier at the government than she is at Madoff. Not only did the government allow him to post bail with "ill-gotten gains," but the Securities and Exchange Commission should have realized Madoff was running a scam, she said.
Congress in January criticized the SEC for missing red flags that should've tipped off the agency.
"I don't think you need a master's degree in investigative techniques to figure out that the first thing you would do is find out: Did he make buys? Did he make sells? What's going on here," Weinberg said. Watch another victim call Madoff 'deluded' »
Madoff had been free on a $10 million bail following his December arrest, but the judge Thursday ordered him remanded to jail.
Fifty courtroom seats were reserved for some of the estimated 4,000 victims of Madoff's Ponzi scheme. Some placed all their savings in Madoff's firm.
Goldworth said he wanted to attend Thursday's hearing for "closure, to see this through."
"It's turned my life inside-out, upside-down, just really have changed everything," he said.