Every day, these men and women of the Foreign Affairs Cybersecurity Center sift through five terabytes of information -- the data equivalent of about 85,000 hours of music or 1.5 million pictures -- looking for abnormalities that could indicate an attempted intrusion into the State Department's vast communications network.
"We're quite proud of the place," said Lonnie Price, the Assistant Director for Cyber and Technology Security in the State Department's Diplomatic Security Service, who hosted reporters for a rare tour of the facility this week.
With so much data to monitor, finding potential intrusions may seem like the proverbial search for the needle in the haystack. But in today's increasingly connected world, both the haystack and the number of needles are growing.
"Cyber is a hot topic," Price noted in his briefing to reporters, "and it's only going to get more so."
"Aging and obsolete technology systems"
But it's not just the prevalence of attacks that the staffers need to worry about. The GAO report also found that the State Department "relies on several aging and obsolete technology systems, which require significant resources to operate and create challenges to ensuring information security."
Put simply, the State Department's cyber infrastructure is something of a dinosaur, and it has earned the scorn of lawmakers and civil servants alike for being outdated, even by government standards.
It quickly caught the attention of Secretary of State Rex Tillerson, a former oil executive, who has made it a priority to reorganize the State Department bureaucracy to run more efficiently.
In an email to employees in September
, Tillerson said the bureau's antiquated technology was a major source of concern for employees in a recent agency-wide survey.
"Secretary Tillerson is absolutely committed to modernizing and innovating within the State Department," said Price, adding that Tillerson is "driving us all very, very hard toward the greater good."
In particular, the State Department is looking toward cloud migration as a way to streamline data accessibility at its missions. By moving data to a remotely accessible server, or "cloud," the State Department would be able to better access and share data at posts around the world. And given the agency's global reach, it's a surprisingly late adopter of the technology.
In 2011, the Obama administration published a government-wide cloud computing
strategy, outlining how the technology could "significantly help agencies grappling with the need to provide highly reliable, innovative services quickly despite resource constraints."
Several branches of the US military have since moved toward cloud migration, as have the departments of homeland security and veterans affairs. These agencies have awarded multi-million dollar contracts to tech giants like Microsoft and IBM.
Price said the State Department is currently considering several potential commercial providers, but wouldn't say which companies have made the short list.
"We want to make sure that they are completely vetted and can handle all of the things -- all of the protective care that we provide our data on premises," said Price, who calls himself "a big fan" of the technology.
"I am completely sold on it," he said, but cautioned that the migration will take time.
"There will be a transition, probably over multiple years," he suggested. "You don't get rid of your legacy infrastructure on premises overnight. ... So we're going to have quite a bit of time to evolve our processes too."
"Spearphishing is a plague upon all of us"
But even the most sophisticated systems are vulnerable to attack if its users aren't on alert. "Spearfishing" scams, in which an email is designed to coax a user into clicking, can put systems at risk.
"Spearphishing is a plague upon all of us here in the US government and beyond," Price said. "It is an extremely effective way to compromise a network, even as well defended as ours."
"It only takes one of these people to click on a link and we have hundreds of man hours of work ahead of us," he added, "if not millions of dollars in damage repair."
That's why the department is fighting spearfishing from all ends, filtering out as many suspicious emails as possible before they hit an employee's inbox, and raising awareness of the problem by conducting "spearfishing exercises" -- sending out emails that look like phishing emails, then tracking how employees respond.
And apparently the fake emails can be quite convincing.
"I have to say -- rather embarrassing for myself -- I received one of these spearfishing messages weeks ago," Price admitted sheepishly.
"It looked just like a LinkedIn -- a person's photo, they want to connect to me," he recounted. "And it has two choices: It says, do you want to connect with this person, or if you don't know them, do you want to see their bio? And I said, 'well, I'm not stupid. I want to see their bio!'"
"And then the immediate message that says you've been phished and that starts the remedial training," he continued. "And I was embarrassed. I've been an information security professional for 30 years, and I was fooled."
Social media, the "soft target"
There's also a growing understanding at the FACC that social media and private email accounts can be used to indirectly target government agencies or officials.
It's a danger that was highlighted on a massive scale during last year's presidential election, when Hillary Clinton's campaign chairman, John Podesta, fell victim to phishing scheme that targeted his Gmail account.
Podesta -- who did not work for the government at the time, but communicated with important figures in government and politics -- clicked a link in an innocuous-looking email, purporting to be a password-reset prompt from Google.
The email was actually a trap set by hackers, who stole tens of thousands of Podesta's emails and provided them to the website Wikileaks.
"One of the things that we're seeing on the rise is personal accounts are becoming more and more popular because they're soft targets in that they're lightly defended and there's a high return on investment," Price revealed, "whether it's just personal stuff, or whether people are taking work home or whatever."
To counter this potential threat, the State Department offers support to its staffers for any issues they might be experiencing with their personal email accounts or social media platforms, in hopes these employees will raise potential incursions with the agency.
"Even if it's their personal account," said Price, "we let them know we're here as a resource. And it is paying off."