Washington (CNN) -- The Justice Department will no longer pursue mandatory minimum sentences for certain low-level, nonviolent drug offenders, Attorney General Eric Holder said Monday, noting the nation is "coldly efficient in jailing criminals," but that it "cannot prosecute or incarcerate" its way to becoming safer.
"Too many Americans go to too many prisons for far too long, and for no truly good law enforcement reason," Holder told the American Bar Association's House of Delegates in San Francisco.
He questioned some assumptions about the criminal justice system's approach to the "war on drugs," saying that excessive incarceration has been an "ineffective and unsustainable" part of it.
Although he said the United States should not abandon being tough on crime, Holder embraced steps to address "shameful" racial disparities in sentencing, the budgetary strains of overpopulated prisons and policies for incarceration that punish and rehabilitate, "not merely to warehouse and forget."
Holder invoked President Barack Obama, saying the two had been talking about the issues and agreed to try to "strike a balance" that clears the way for a "pragmatic" and "commonsense" solutions to enhance public safety and the "public good."
The centerpiece of Holder's plan is to scale back prosecution for certain drug offenders -- those with no ties to large-scale organizations, gangs or cartels. He said they would no longer be charged with offenses that "impose draconian mandatory minimum sentences."
They now "will be charged with offenses for which the accompanying sentences are better suited to their individual conduct, rather than excessive prison terms more appropriate for violent criminals or drug kingpins."
The changes are effective immediately.
Lessening the use of mandatory minimums -- sentences that require a "one-size-fits-all" punishment for those convicted of federal and state crimes -- could mark the end of the tough-on-crime era that began with strict anti-drug laws in the 1970s and accelerated with mandatory minimum prison sentences and so-called three-strikes laws.
The attorney general linked the effort to rethink mandatory minimum sentencing for drug crimes to key issues.
Holder said "unwarranted disparities are far too common" in the criminal justice system, reminding his audience that Obama alluded to some of the issues in remarks he made after the acquittal of George Zimmerman in the death of Trayvon Martin last month, giving voice to African-American concerns that "there is a history of racial disparities in the application of our criminal laws -- everything from the death penalty to enforcement of our drug laws."
Holder said the nation "must confront the reality" that once "people of color" are in the criminal justice system, they "often face harsher punishments than their peers." He called it "unacceptable," "shameful" and "unworthy" of the U.S. legal tradition.
Holder said the U.S. prison population has grown by almost 800% since 1980, and federal prisons are operating at nearly 40% above capacity.
"Even though this country comprises just 5% of the world's population, we incarcerate almost a quarter of the world's prisoners. More than 219,000 federal inmates are currently behind bars," he said, noting that almost half are serving time for drug-related crimes and have substance abuse problems.
Moreover, he said 9 million to 10 million more people cycle through America's local jails each year. And roughly 40% of former federal prisoners -- and more than 60% of former state prisoners -- are rearrested or have their supervision revoked within three years after their release.
It was not immediately clear Monday whether Holder's announcement would have any impact on people already in prison.
Economic, social burden
Holder said overcrowding at the federal, state and local levels is "both ineffective and unsustainable." He said it imposes a significant economic burden -- totaling $80 billion in 2010 alone -- and it comes with "human and moral costs that are impossible to calculate."
Legislation to lessen the use of mandatory minimums, Holder said, would ultimately save the United States billions.
Although Obama administration officials say the changes they are pursuing will not require congressional approval, some unlikely pairs of lawmakers have united to push for criminal justice changes.
Democratic Sen. Patrick Leahy of Vermont and Republican Sen. Rand Paul of Kentucky have worked together to allow judges to depart from mandatory minimum sentences when circumstances merit. Democratic Sen. Richard Durbin of Illinois and Republican Sen. Mike Lee of Utah have undertaken similar efforts.
In recent years, there has been a rise in support among conservatives for reforms to the criminal justice system. While more flexible approaches to crimes have long held support among liberal Democrats, the fear of being tarred as weak on crime by Republican opponents has long caused moderate Democrats, particularly those running for president, to avoid the issue.
In addition to changes to mandatory minimums, Holder called for expanding the use of "compassionate release" from jail for those who "pose no threat to the public."
He also said the Justice Department is taking steps to identify practices for enhancing the use of drug treatment and community service programs as alternatives to jail.
Holder also said he has asked federal prosecutors to develop new guidelines for determining when federal charges should be filed and when they should not.
"I've also issued guidance to ensure that every case we bring serves a substantial federal interest and complements the work of our law enforcement partners," he said.
Holder additionally directed prosecutors to create comprehensive anti-violence strategies for badly afflicted areas.
The American Civil Liberties Union praised Holder's approach Monday, calling it an important step toward ending federal prison overcrowding and creating a "fairer criminal justice system."
Laura Murphy, director of the ACLU's Washington legislative office, said in a news release, however, that although Holder's announcement "is an important first step," Congress also must act to change laws that "lock up hundreds of thousands of Americans unfairly and unnecessarily."
Michael Mukasey, a former judge and an attorney general under President George W. Bush, said he is not a fan of mandatory minimums, but he does not support what Holder is doing.
"I generally agree with the goal of getting rid of mandatory minimums," said Mukasey. "But the way to do that is to pass a law," he said in an interview on CNN's "The Lead with Jake Tapper."
CNN's Carol Cratty and Jessica Yellin contributed to this report.