Skip to main content

How did Google lose, and find, all those e-mails?

John D. Sutter
Google stores e-mail data all over the world in huge warehouses of information, like this one in Oregon.
Google stores e-mail data all over the world in huge warehouses of information, like this one in Oregon.
  • Google loses -- and then finds -- some users' e-mails
  • The company keeps backups on servers around the world and on physical tape
  • Experts say companies like Google are better-equipped to care for data than average user

(CNN) -- Tens of thousands of Google e-mail users got a shock early this week: All of their e-mails and contacts disappeared.

Google said Monday night that it was in the process of restoring all of these messages, however. "We're very sorry," the company said in a blog post.

The burning question here is, how did Google lose all of these e-mails, and how was the company able to get them all back, if they in fact were lost?

And, underlying that question: Should you trust Google? Is the company well-suited to safeguard all of this sensitive information, which is essential to both business and personal life? And what guarantees do you have that the company will keep your data safe, given that Gmail is a free service?

Let's start with the first question.

Google says it lost the e-mails because of a glitch in a software update that it was in the process of installing across its computer servers.

If you know much about Google, you probably know that the company stores multiple copies of everything in data centers -- huge warehouses full of computers -- all over the world, in secret locations. So that e-mail you sent your grandma last night may be hiding out in Asia, in Europe and on the east and west coasts of the United States.

Google's blog post addresses this point:

"I know what some of you are thinking: How could this happen if we have multiple copies of your data, in multiple data centers? Well, in some rare instances software bugs can affect several copies of the data. That's what happened here. Some copies of mail were deleted, and we've been hard at work over the last 30 hours getting it back for the people affected by this issue."

So it was a software glitch. Makes sense. But how did the company recover all of this data if the glitch traveled so widely so quickly?

Answer: Google still stores e-mail data on tape.

Yes, tape. Like, essentially the technology behind the cassette. Google doesn't specify what kind of tape it's using exactly, but as Seth Weintraub notes on a blog on the CNN partner site, using tape is a mess.

Weintraub estimates Google would need 200,000 tapes to make a single backup of every Gmail account. "That is a stack of tapes four kilometers high to back up Gmail," he writes. "Ouch."

The blog Data Center Knowledge, however, says tape is still a worthy method of backing up data, even if it seems obsolete to outsiders:

"Even today, tape has two significant advantages over other media: cost and portability. Unfortunately, these two advantages outweigh the more significant (logically speaking) disadvantages of tape media: fragility, replacement rate, failure rate, vulnerability to theft, and unencrypted data storage."

Google puts it this way:

"To protect your information from these unusual bugs, we also back it up to tape. Since the tapes are offline, they're protected from such software bugs. But restoring data from them also takes longer than transferring your requests to another data center, which is why it's taken us hours to get the e-mail back instead of milliseconds."

OK, so that's it for the logistics. But what should you take away from this incident, which, it should be noted, affected only 0.02% of Gmail users? (Google says it has hundreds of millions of Gmail users, which means at least 20,000 people were affected by this outage).

Many data storage experts say companies like Google -- and competitors like Microsoft, Yahoo and Dropbox, all of which store data in "the cloud" -- are usually better equipped to take care of data than normal people like us, who tend to make one backup -- at most -- of the data on our hard drives.

Google, Yahoo and the others tend to make multiple copies, and they store them all over the world so that if one data center burns down or floods, there are plenty of copies in far-and-away locations that will be unaffected.

By contrast, if a person's apartment building were seriously damaged, he or she might lose a laptop and the hard drive that's used as backup. Most of us don't send a copy to India just in case, much less store e-mail on tape.

But there's no 100% guarantee, really.

Google and the others have "terms of service" agreements with users that guarantee a certain amount of up-time, usually 99.9%.

"People expect email to be as reliable as their phone's dial tone, and our goal is to deliver that kind of always-on availability with our applications," Google's Matthew Glotzbach wrote in a January blog post.

Last year, Google says, it met its goal of being up 99.9% of the time:

"In 2010, Gmail was available 99.984% of the time, for both business and consumer users. 99.984% translates to seven minutes of downtime per month over the last year. That seven-minute average represents the accumulation of small delays of a few seconds, and most people experienced no issues at all."

Users still freak out, however, if Gmail or other free online services go down for even a short period of time. And that's probably why Google and the others go to such lengths to avoid losing user data completely.

It's difficult to compare free e-mail systems to each other because data about uptime isn't always available. Both Microsoft, which owns Hotmail, and Yahoo declined to tell CNN how often their services go down, or what percent of the time these e-mail systems are up and online.

To keep tabs on how well Google is doing at keeping its e-mail service up and running, you can check the company's "App Status Dashboard" site.


Most popular Tech stories right now