It has to do with the tiny cones in the back of our eyeballs that perceive colors in a slightly different way depending upon our genes.
"Why do some people love cilantro and others say it tastes like soap? Why do some people have perfect pitch and others are tone deaf? It's the same with vision — our sensory apparatus is fine tuned," says Dr. Julia Haller, the ophthalmologist-in-chief at Wills Eye Hospital in Philadelphia.
The cones in our retinas — the fine layer of nerve tissue that lines the back of our eyes — detect the blue, green, and red in an image. The cones and your brain mix those colors to make other colors.
"Ninety-nine percent of the time, we'll see the same colors," Haller says. "But the picture of this dress seems to have tints that hit the sweet spot that's confusing to a lot of people."
The very top section of the dress appears gold to some people, but black to others.
This makes sense to Haller.
"One of the typical color confusions we see is blue/yellow," she says. "So perhaps in this dress, the black has a bit of blue and the gold has a bit of yellow."
As retinal specialists met Friday in Scottsdale, Arizona, at the annual meeting of the Macula Society, they swapped theories about why people's cones and brains process the dress's colors so differently.
One theory has to do with evolutionary biology. Primates, including humans, have developed an excellent sense of color, so we're able to, for example, distinguish the yellow fruit hanging on the green tree.
To determine the color of that fruit, it's often necessary to factor out colors in the light around it that might give the fruit a different and misleading tint. There are likely slight variations in how our brains do this automatic filtering.
"Your brain is constantly estimating the color of the light that's falling on the object and factoring that light out," said Wallace Thoreson, a professor of ophthalmology at the University of Nebraska Medical Center. "Each of us makes slightly different unconscious assumptions."
Haller said the Macula Society's meeting was "totally derailed by the dress" as doctors skipped sessions to stand out in the hallway and compare each other's color judgments.
Some of them even think the dress could possibly help advance medical science and hopefully help people with vision problems.
Dr. Emily Chew, an ophthalmologist at the National Institutes of Health, said in more than thirty years of research she's never seen an image where people, all with normal vision, have had such dramatically different responses.
"It would be interesting to use this image to learn more about the pathways of how we understand color," said Chew, the deputy director of the division of epidemiology and clinical applications at the National Eye Institute.
She said it was impressive how "crowd sourcing" identified the dress.
"It would be really hard to create this image," she said. "It's serendipitous. I've never seen anything like this. It's really unbelievable."
CNN's Nadia Kounang contributed to this report.