(CNN) -- As he watched the presidential debates, Bretton Holmes was irritated. But it wasn't the candidates who were getting to him, it was hearing independent voters and undecided voters lumped into the same group.
The 35-year-old from Phoenix has registered as an independent since he was 18. He has voted for Democrats and Republicans over the years.
The biggest misconception he hears is this: "If you're registered as an independent, that must mean you're undecided," he said.
"That's just an opinion that happens to be very incorrect," he said. "Being independent has nothing to do with being undecided."
Living in a two-party country can be tough for this group of voters, but there are more independents these days than ever.
Thirty-eight percent of Americans identify themselves as independents, according to a 2012 study on party identification by the Pew Research Center for the People & the Press.
"The proportion of independents now equals its highest level in 70 years," said a different Pew study from 2009. "Owing to defections from the Republican Party, independents are more conservative on several key issues than in the past."
Being called an undecided voter irks "independent thinker" Holmes, as he already has made his decision. "In my case, it was already clear who I wanted to vote for," he said. He cast an early ballot for President Barack Obama.
CNN asked independents why they refuse to align with a party and heard from more than 100 people. A variety of themes arose, from not wanting to be labeled to disgust with political rancor to feeling that being independent is the "spirit of America."
'I don't like being labeled'
Omekongo Dibinga of Washington has consistently voted Democrat but explains that he's always identified himself as an independent.
He's never voted for a Republican because "I haven't agreed with any of the candidates' values, but I reserve the right to vote for a Republican if their values match mine," he said.
Growing up in an impoverished neighborhood in Boston where "violence was a problem," Dibinga faced stereotypes that he has worked to overcome as a diversity consultant and motivational speaker.
"I don't like being labeled. I've been labeled a lot of things in my life," he said.
Jennifer Cummins, a moderate independent, has a similar problem with siding with just one party.
"It's mostly the label," she said. "If you say you're a Democrat, that must mean you are a left-wing liberal with no personal responsibility. If you say you are a Republican, you must be a right-wing millionaire who doesn't care about others."
The Frankfort, Kentucky, voter feels a "lack of respect" toward independents like her and wishes there was more air time given to independent candidates.
"I think it's a travesty that the only 'limelight' that exists is with either Republican or Democratic when there are other options on the ballot with better opinions, much better track records, and a real solution," she said.
Voting for the 'lesser of two evils'
Roger Cantillo, 37, identified more with the Democratic Party when he was younger, but he started considering himself an independent in 2008.
"The 2008 presidential race is when I really started getting involved in politics and trying to understand what's going on in Washington. It's just unfortunate that there's a lot of gridlock, and people are playing both sides," he said. In the last few elections, he voted for Republican George W. Bush, Democrat John Kerry and, most recently, Obama.
While Cantillo's "progressive values" indicate he leans more to the left, Cantillo said he finds faults with both parties. He will be voting for Obama on Election Day, but he's not enthusiastic about it.
"I'm picking the lesser of two evils come this election."
It's the 'spirit of America'
Raised as a Southern Democrat who switched to voting Republican in 1999, Mary Helen Yarborough has voted for presidential candidates from both parties equally. She has identified as a "confused" independent for 10 years, she said.
GOP presidential candidates Rick Perry and Newt Gingrich excited the right-leaning independent in the primaries, and she "flirted" with the idea of voting for Obama back in July. Ultimately, she says, her vote will go to Mitt Romney next week.
But the opinionated South Carolinian doesn't want to be tied down to a party. "I don't want to be controlled," she said. "I feel better as an independent. I feel like it's a more honest position."
Voting independent just feels "more American," she said. "America was born on the theme of independence, and I find that being a committed independent is therefore truer to our national pride."
Betty Faller-Pearson, 66, voted for Obama in 2008 and Bush in 2004. She agrees with Yarborough's line of thinking.
"I don't vote party lines because I don't always agree with either party, Democrats or Republicans," she said. "I'm not liberal, I'm not conservative. I'm independent and can make my own decisions."
For the Las Vegas resident, being an independent goes back to America's roots. "I am an independent voter because I believe in the spirit of America and how and why it was founded," she said.
Raised Republican in Texas and now married to a Democrat from New Jersey, Jim Mitchem's flavor of independence stems from a similar tree.
"I've never been one for dogma and don't feel like any 'side' could represent my free will well enough to go straight ticket," he said.
The 44-year-old has voted for Ronald Reagan, Bush, Ross Perot (twice), Al Gore and Obama. He said it's "too restricting" to affiliate with one party. "In the spirit of independence by Americans who broke away from political dogma in the 1700s, I'm proud to be unaffiliated."
Lost between parties
Brian Pigg confesses he feels "lost" politically.
Growing up in the Reagan era, the 44-year-old holds the Republican beliefs of smaller government and fiscal responsibility, he said. As a veteran, he supports having a strong military.
But his concerns with the increasing U.S. debt and George W. Bush's presidency made him reconsider the way he voted, he said.
"Years ago, I began to feel disowned by my party, as this debt didn't start in '08 (not that it isn't getting worse)," he wrote on CNN iReport. The "rise of the religious right" was the final straw.
What's a man without a party to do? At first he thought about abstaining from voting.
"I haven't heard a single candidate worth voting for, not in local, state, or federal elections," he said. "They all seem to say whatever they think will get them votes without actually being nailed to anything."
The Grandview, Missouri, resident admits he wasn't going to vote at all this year until controversy arose around U.S. Senate candidate Todd Akin's comments on rape. "Now I'm going just to vote against him. How sad is that," he asked.
Be 'free thinkers'
But Holmes, the voter who emphasized the difference between independents and undecideds, asserts that voting as "free thinkers" is the answer.
"Everyone has the ability to think for themselves and question what other people are saying if they don't think it's right," he said. "Speak up!"