The risks -- and benefits -- of letting algorithms judge us

Chinese Internet users may be judged by what they do on line under a new system called Sesame Credit.

Story highlights

  • China is considering a new "social credit" system, designed to rate everyone's trustworthiness
  • It sounds ominous, but has a lot in common with automated systems already in use in the U.S., Bruce Schneier writes
  • We can get the benefits of automatic algorithmic systems while avoiding the dangers, Schneier says

Bruce Schneier is a security technologist and chief technology officer of Resilient Systems Inc. His latest book is "Data and Goliath: The Hidden Battles to Collect Your Data and Control Your World." He blogs at schneier.com and tweets @schneierblog. The opinions expressed in this commentary are solely those of the author.

(CNN)China is considering a new "social credit" system, designed to rate everyone's trustworthiness. Many fear that it will become a tool of social control -- but in reality it has a lot in common with the algorithms and systems that score and classify us all every day.

Human judgment is being replaced by automatic algorithms, and that brings with it both enormous benefits and risks. The technology is enabling a new form of social control, sometimes deliberately and sometimes as a side effect. And as the Internet of Things ushers in an era of more sensors and more data -- and more algorithms -- we need to ensure that we reap the benefits while avoiding the harms.
Bruce Schneier
Right now, the Chinese government is watching how companies use "social credit" scores in state-approved pilot projects. The most prominent one is Sesame Credit, and it's much more than a financial scoring system.
    Citizens are judged not only by conventional financial criteria, but by their actions and associations. Rumors abound about how this system works. Various news sites are speculating that your score will go up if you share a link from a state-sponsored news agency and go down if you post pictures of Tiananmen Square. Similarly, your score will go up if you purchase local agricultural products and down if you purchase Japanese anime. Right now the worst fears seem overblown, but could certainly come to pass in the future.

    Algorithmic scoring is everywhere

    This story has spread because it's just the sort of behavior you'd expect from the authoritarian government in China. But there's little about the scoring systems used by Sesame Credit that's unique to China. All of us are being categorized and judged by similar algorithms, both by companies and by governments. While the aim of these systems might not be social control, it's often the byproduct. And if we're not careful, the creepy results we imagine for the Chinese will be our lot as well.
    Sesame Credit is largely based on a U.S. system called FICO. That's the system that determines your credit score. You actually have a few dozen different ones, and they determine whether you can get a mortgage, car loan or credit card, and what sorts of interest rates you're offered. The exact algorithm is secret, but we know in general what goes into a FICO score: how much debt you have, how good you've been at repaying your debt, how long your credit history is and so on.
    There's nothing about your social network, but that might change. In August, Facebook was awarded a patent on using a borrower's social network to help determine if he or she is a good credit risk. Basically, your creditworthiness becomes dependent on the creditworthiness of your friends. Associate with deadbeats, and you're more likely to be judged as one.
    Your associations can be used to judge you in other ways as well. It's now common for employers to use social media sites to screen job applicants. This manual process is increasingly being outsourced and automated; companies like Social Intelligence and Evolv automatically process your social networking activity and provide hiring recommendations for employers. The dangers of this type of system -- from discriminatory biases resulting from the data to an obsession with scores over more social measures -- are too many.
    The company Klout tried to make a business of measuring your online influence, hoping its proprietary system would become an industry standard used for things like hiring and giving out free product samples.

    Local banker's judgment vs. computer crunching

    The U.S. government is judging you as well. Your social media postings could get you on the terrorist watch list, affecting your ability to fly on an airplane and even get a job. In 2012, a British tourist's tweet caused the U.S. to deny him entry into the country. We know that the National Security Agency uses complex computer algorithms to sift through the Internet data it collects on both Americans and foreigners.
    All of these systems, from Sesame Credit to the NSA's secret algorithms, are made possible by computers and data. A couple of generations ago, you would apply for a home mortgage at a bank that knew you, and a bank manager would make a determination of your creditworthiness. Yes, the system was prone to all sorts of abuses, ranging from discrimination to an old-boy network of friends helping friends. But the system also couldn't scale. It made no sense for a bank across the state to give you a loan, because they didn't know you. Loans stayed local.
    FICO scores changed that. Now, a computer crunches your credit history and produces a number. And you can take that number to any mortgage lender in the country. The