(CNN) -- Marijuana makes you slow and raises the risk of addiction, memory problems and cognitive functions, some health agencies warn.
But its enthusiasts counter that in addition to fueling creativity, pot is a natural herb with health benefits, relieving pain and improving quality of life for chronically ill patients.
Medical use of marijuana is legal in 14 states and the District of Columbia. On Tuesday, California voters rejected a ballot initiative that would've made their state the first to legalize recreational marijuana use.
The vote on Proposition 19 was being closely watched as the Golden State is often seen as a bellwether. It was the first state to authorize medicinal marijuana in 1996.
Prop 19 would have allowed people 21 and older to possess, cultivate or transport up to 1 ounce of marijuana for personal use.
The intense debate about marijuana legalization has caused science to get lost in the rhetorical shuffle, said Alan Budney, professor of psychiatry and a research scientist in the Center for Addiction Research at the University of Arkansas.
"The government wants to put things out in sound bites," he said. "The pro-marijuana camp puts out sound bites. It's all or nothing."
The medical evidence on marijuana is "not black and white," Budney said. "When people say one thing, you can bet it's somewhere in the middle. It's all pretty gray."
Marijuana is the most widely used illicit drug in the United States, according to the Office of National Drug Control Policy.
Proponents for legalizing its use often say that marijuana is far less damaging than alcohol, which is legal.
"Have you ever heard of a fight in a bar caused by smoking marijuana? Or killing someone because of marijuana?" asked Dr. Lester Grinspoon, a psychiatrist and retired professor at the Harvard Medical School who has published several books and articles in drugs and drug policy.
"You probably never will, because number one, alcohol enhances aggressiveness, marijuana does the opposite," he said. "People don't want to punch anybody. They want to be friendly and left alone. It doesn't cause the violence. It's a safer recreational drug because of that."
In pop culture, marijuana is often portrayed as an innocuous drug. In a show of support for Prop. 19, actor Zach Galifianakis appeared to light up a joint on HBO's "Real Time" with Bill Maher on Friday.
Medicinal marijuana is used to stimulate appetite, relieve pain and reduce nausea.
One of the issues is that people believe how marijuana affects them is how it affects everyone, said Dr. Nora Volkow, director of the National Institute on Drug Abuse.
"They base it on their own experiences, and it is possible you can smoke and have no ill effects. It's also possible you can smoke until age 100 and have no ill effects. Whether you're vulnerable to toxic effects of smoking or marijuana is dependent on unique genes."
"We don't know what genes we're carrying -- it's like playing roulette," she said. "If you have a genetic risk for addiction or higher sensitivity, all of those can result in serious adverse effects of smoking marijuana."
The drug could increase risk of psychotic episodes and impair short-term memory, motor coordination and decision-making, Volkow said.
Marijuana contains more than 400 chemical constituents, and its main psychoactive ingredient is tetrahydrocannabinol, THC for short.
It binds to cannabinoid receptors in the brain in areas that influence pleasure, memory, thought, concentration, sensory and time perception, appetite, pain and movement coordination.
This could lead to the feeling of calmness, euphoria and creativity.
"It's a great antidepressant, and you feel good," said Grinspoon, a marijuana supporter. "It is a slight alteration of consciousness. It does open up one's mind and new ideas come to mind."
Grinspoon said he had staked his professional career on the notion that "cannabis is the least toxic psychoactive drug known to man."
Unlike alcohol, there has not been any deaths from overdose of marijuana, he said.
"It's also much less harmful in so many other ways," he said. "There are things like cirrhosis from alcohol, but there's no such organ damage with cannabis."
Volkow said the impression that marijuana is harmless is wrong. Along with effects in the brain, she said chronic marijuana smoking could inflame lung cells, increasing the risk of chronic bronchitis.
"There is no scientific evidence that it's benign," she said. "Indeed, there's substantial evidence of adverse effects. What are consequences of marijuana? Number one, of the people who get exposed, 10 percent will become addicted. Despite what everyone wants to believe, the evidence is there."
Cannabinoid receptors are found in the cerebellum, an area of the brain that's indispensable for coordination. This has raised concerns that legalizing marijuana would increase the number of people driving stoned.
People shouldn't drive after using marijuana because of safety reasons, Grinspoon said, but they're not likely to cause the same problems caused by drunken drivers.
"If you see someone driving on the right side of the road, driving timidly ... that's someone driving stoned," he said. "They drive timidly, not to justify that. It's just the opposite of the alcoholic who gets arrogant on the road."
"They have no urgency about them; that might be part of their driving slow."
Budney said that cannabis is useful in some conditions as science continually evolves.
"We know marijuana affects the system; we don't know how much," he said.
"Heavy use of marijuana, just like any substance, is going to interact with brain," he said. "Something you smoke is going to have an effect on the lungs. If you do too much of it, how serious are those effects? We know it affects memory, complex tasks."
According to a USC/Los Angeles Times Poll, 51 percent of likely voters said they would vote against legalization while 39 percent indicated they supported the measure.