Often mistaken as a cancer sufferer, strangers will ask her how long she's been stricken with the disease.
"Just last week, I was coming out of the women's toilets and this woman goes 'sorry, wrong toilets,'" says the 32-year-old rugby player.
"A year ago, I would have walked away in shame but this time I was like 'no, you're alright.' And I'll get 'sorry mate' a lot as though I'm a bloke, but it never gets any easier to take. I'm just a human being at the end of the day.
"But I'd say don't judge a book by its cover. We all have things we don't like about ourselves and we choose to show that or not. It's just that mine is always on show."
Fisher has the condition alopecia, which results in hair loss, and also struggled with anorexia in her teenage years, but is remarkably and refreshingly honest about both issues.
"I am what I am at the end of the day so I'm straight and up front," she says. "I'm like Marmite, you either like or you don't like me."
She remains one of the most explosive players in the game, predominantly in the sevens format in which she represented Great Britain at the Rio Olympics and is in England's squad for the penultimate round of the World Rugby Women's Sevens Series
in Langford, which starts on May 27.
Fisher inspires off the field too, acting as an ambassador for Alopecia UK and living by the motto which is written on her website: "work hard, be brave, believe in yourself"
But Fisher is the first to admit her self-belief has not always been rock solid. A keen sportswoman growing up, rowing was her first sporting foray of note at a time when her parents were divorcing.
"I used to row a lot when I was younger," she recalls of her teenage self. "I was in the single scull and double scull and part of it was making the weight category. My parents went through a divorce at the time and I lost contact with my Dad.
"I had no control over my life. It was all so wrong and I was angry inside, and the only thing I could control was to not eat."
It is something, she says, that remains with her and, if her mood slips or there is sadness in her life, she stops eating.
But it is a condition she now has a grip on, having struggled with it for a solid 18 months.
"I remember my Mum seeing me in the bath and saying how skinny I was, and I knew she was right," she adds.
"I knew I needed help. I cried my eyes out some mornings and would say to myself 'this is not right'. I just didn't know how to control what I had in my head."
Passed around from doctors to nutritionists and back again, it was a chance encounter with one particular doctor that changed her perspective.
Fisher says: "This doctor said to me 'you know to be an athlete you have to eat'. I don't know who he was or his name but he changed me and my career. Hopefully my voice can make a change in just one person."
For as long as she could remember, Fisher wanted to be an elite sportswoman. The doctor's words transformed her life and allowed her to go on and succeed in various sports. First, from 2006 to 2009, in the British bobsleigh set-up, where she was second in the national trials in 2008 and then sixth in the junior world championships.
But the lure of rugby was strong -- a sport she had played before bobsleigh. She was part of the squad which won the 2014 Rugby World Cup and remains a regular on the World Cup Sevens circuit.
Rio was supposed to be the high
but, following a semifinal defeat by New Zealand and missing out on bronze to Canada, she has struggled in the intervening months.
At the time, she wrote a heartfelt message on social media of her Olympic journey, from the anorexia and alopecia to, more recently, injury setbacks. Even now, she has not fully come to terms with it.
"I'm still in that post-Rio dip," she confesses. "The result was what it was and I came away thinking 'is this it?' with everything that went into it. You don't know where to place yourself and I want to put things to bed, but I can't."
Partly, that is down to the fact that England are in a transition period, blooding new players in order to prepare for next year's Commonwealth Games on the Gold Coast, a driving factor for Fisher.
"I know it's about getting the balance but it can be frustrating," she says. "But it's all about peaking for then and maybe we didn't peak quite right for Rio."
Of her family life, I ask if it is now more settled and she jokes: "It's like an episode of Eastenders [a UK television series]; actually it feels like we're on episode 20 of series five!"
But the support of family, friends, teammates and support staff have transformed Fisher as a person and a player.
She says she wouldn't wish her battles on anyone, but she is not looking for sympathy, merely to inspire those around her and those watching from afar.
"I'll be on holiday or I'll bump into a kid, and then they'll come up to me and say 'you're so brave'. You just want to help or inspire one person. That's the goal," she says.