Dickens, 94, died at a hospital in Nashville, according to media reports.
"It is with a heavy heart that I say goodbye to my hero and friend today. I loved you Jimmy," Brad Paisley wrote on Twitter.
Dickens joined the Opry, the showcase for country music, in 1948.
"The Grand Ole Opry did not have a better friend than Little Jimmy Dickens," Opry Vice President and General Manager Pete Fisher said in a statement. "He loved the audience and his Opry family, and all of us loved him back. He was a one-of-kind entertainer and a great soul whose spirit will live on for years to come."
Dickens -- known for his diminutive stature, colorful rhinestone-studded jackets and keen sense of humor -- had several novelty hits, beginning in 1949. "May The Bird Of Paradise Fly Up Your Nose," "Out Behind the Barn," and "Take An Old Cold Tater (And Wait)" are among his fans' favorite tunes. Another is "I'm Little, But I'm Loud."
The 4-foot-11-inch artist was inducted into the Country Music Hall of Fame in 1983. He is survived by his wife, Mona Dickens, and two daughters.
"I am deeply saddened. We have lost a precious treasure," tweeted Charlie Daniels.
The Opry's news release said plans were still being formed for a funeral, as well as a public viewing and service.
Dickens was born in Bolt, West Virginia, the oldest of 13 children. In 2007 he told West Virginia Public Broadcasting that while his dad was a coal miner and the family was poor, they were never hungry.
"Nothing easy about coal mining for a living but we all got by," he said. "There was more love in our family than the money."
He said his grandparents raised him. His mother and uncles played guitars and banjos but none of them played professionally, he said.
He got his start on a radio station in Beckley, according to a biography on the Hall of Fame website. He had a $12 guitar that he carried in a pillowcase and he got to sing a song or two each week.
Eventually he'd go out and sing with the stars of the station, he said. He was still in high school at the time.
He moved on to a station in Cincinnati, where he was doing a morning show when country star Roy Acuff came for a visit in the 1940s. Acuff invited Dickens to sing a song in his show, which the star liked enough to help get Dickens an invitation to sing at the Opry even before he had a record.
"I went down there and did that and it worked for me," Dickens said.
He said he wanted to be remembered for his honesty with his fans and coworkers and for his desire to always get better as a performer.