Now that's starting to change, as some of the biggest names in finance adjust their businesses to reflect growing demand for impact investments.
"It's a $250 billion market and it's growing fast," said Michael Baldinger, a 30-year industry veteran who joined UBS (UBS) Asset Management as head of sustainable and impact investing -- a new role -- just over a year ago. "It might really be a game-changer for the finance industry."
Banks and asset managers are thinking seriously about how to cater to investors who want a measurable social or environmental return on their investment as well as capital gains, stock dividends or interest payments.
"In the past you sold products to your client, now you empower your client to create a desired impact," Baldinger said. "As an industry we've had to rethink everything we do -- impact and sustainability is the Silicon Valley of finance and we want to be the Google."
Impact investments were once the preserve of the super rich. Individuals and families would come together to identify promising opportunities to make money and do good at the same time.
But that's changing fast, too.
"76% of millennials see investment decisions as a way to express their social, political and environmental values," said Jackie VanderBrug, a managing director at U.S. Trust, Bank of America's (BAC) wealth investment management business.
"When three quarters of a population are saying 'this is important,' that's huge," she added.
Before joining U.S. Trust in 2013, VanderBrug helped to start an impact enterprise focused on career development for underprivileged youth. She was instrumental in establishing Good Capital, a social investment fund.
Now she identifies impact investment opportunities, including portfolios with a minimum investment of $12,000 up to fully bespoke strategies for bigger clients.
Barclays (BCS) is another major bank with big ambitions in this area. It launched its Multi-Impact Growth Fund in September.
Damian Payiatakis, head of impact investing at Barclays, said the British bank wants to integrate the ethos of impact investing across everything it does.
"Our interest is not to add impact products to a shelf of investments," Payiatakis said. "Once you start seeing the world that way, it becomes obvious and you ask, 'why not,'" he said.
More work to be done
Still, there may be some way to go before impact investing becomes truly mainstream even as it continues to grow.
"While the idea that the only job of a company is to maximize financial returns to shareholders is still fairly entrenched, I think we have the wind at our backs in the form of our younger generations, who have higher expectations," said Deval Patrick, managing director of Boston-based Bain Capital's Double Impact business, and a former governor of Massachusetts.
"Many of those same individuals are taking a look at their investment dollars through a similar lens and with an expectation of social responsibility, so I expect the field to continue to grow."
For Baldinger at UBS, part of the growth can be attributed to greater transparency about impact investing, which gives the investor a choice.
"In the 1990s, I used to go to Starbucks each morning for my latte and my muffin. Then they started to show the calories next to it and that triggered something in me, so I stopped," he said.
"They created transparency for the customer. You can ignore it, but you have the information and you have a choice."