It is the kind of cyber resource that Hillary Clinton has seized on for her proposed "intelligence surge" to defeat ISIS and other terrorist threats as they recruit adherents and plan attacks using online tools.
Meeting with national security experts in September, Clinton said an area of deep focus was "what it would take to surge our intelligence, to help us detect and prevent attacks before they happen."
In Tuesday's vice presidential debate, Clinton's running mate, Virginia Sen. Tim Kaine, said the pair would work to strike "great partnerships with some of our cyber and intel experts in the private sector so that we can, consistent with constitutional principles, gather more intelligence."
But the reliance on technology deepens legal and ethical concerns that civil liberties groups and other natural allies of the Democratic Party have about law enforcement's use of data and technology, and its monitoring of social media.
The Justice Department's announcement Wednesday of the arrest of a National Security Agency contractor for allegedly stealing classified intelligence only underscored the challenges and sensitivities of boosting such surveillance.
The expanding uses of technology and new capabilities in particular have raised alarm bells at the American Civil Liberties Union.
"There really hasn't been an adequate discussion of this or adequate appreciation in the broader public or attention by legislators for the consequences on our political discourse or our democracy writ large," said Hugh Handeyside, a staff attorney for the ACLU's National Security Project.
He continued, "It's just beginning to dawn on us that just because this capability exists, doesn't mean it should be used."
Predictive analytical tool
The analytic tool, called "graph technology," isn't about sucking up more data; it's about being able to more efficiently sift through it. Experts say it can be used to spot signs of imminent cyber attacks, such as the theft of Democratic National Committee emails or other hacking attempts on voter registration systems in 20 states.
They maintain that graph technology, initially released about seven years ago, would work equally well in hunting down terrorist cells.
"We think in the next few years, tech will be able to help in a much greater way" with national security work, said Mark Testoni, president of SAP National Security Services.
"We've doubled the entirety of digital information in the entire world in the last two or three years," said Testoni, who added that when you apply high-speed analytics like graph technology to this mass of information, "It offers great promise."
"It's not just the traditional data," he explained, referring to social media and data on individuals, but resources like "geospatial information, too."
That includes mapping data and images collected by satellites, drones and aircraft, as well as GPS waypoints, census information and other data tied to specific locations.
"This is the great promise and hope," Testoni said.
Clinton's call for more intelligence capabilities -- human and cyber tools -- to deal with terrorism includes better cooperation between local, state and federal agencies within the US; more robust intelligence sharing among US allies, particularly in Europe; and improved intelligence gathering in the Middle East, including surveillance, reconnaissance and collecting information from both data and human sources.
Jim Lewis, director of the Strategic Technologies Program at the Center for Strategic and International Studies, said that a cyber surge could be crucial as ISIS loses ground in Syria and tries "to slip more into the West."
"The way we used to protect ourselves may not work so well in the future," Lewis said. "In some areas we're going to have to increase what we do, and big data may be a part of that."
The newer capabilities of graph technology can be used to find patterns by looking at how data is connected and the nature of those connections.
For example, an investigator could use a graph to find everyone who went to high school with an individual or shared the same address. It's a very sophisticated version of the technology that Facebook uses to send "friend" suggestions.
Those connections can be used to build algorithms that can spot anomalies or flag certain activities based on previous patterns. If a bank has ever contacted you because of suspicious activity on your account, you've experienced the power of these algorithms.
Experts say that this resource could help identify patterns of activity that preceded past terror attacks or cyber hacks. By programming the software to search data for those patterns, investigators can then look more closely to see whether other hacks or plots are indeed underway.
"We're starting to see strong adaption of the technology" by national security agencies, said Emil Eifrem, CEO of Neo Technology, the San Mateo, California, firm that invented the approach and coined the term "graph technology."
Preventing cyber attacks
While Eifrem said that the technology isn't yet being used in "predictive ways" for national security, he said it could easily help point to imminent cyber intrusions of the kind seen on voter registration systems and the DNC.
Using graph technology, for instance, you could identify the network of IP addresses used in attacks, the central server the addresses link back to and the way these addresses were used.
"What isn't being done yet," said Eifrem, "is that you embed all that in the graph database, so you can start predicting it ahead of time," and say, "based on network traffic that we've seen before, we suspect we're going to see a breach attempt."
The technology "translates identically, completely, into national security" for investigators looking for connections between terrorists or other plotters, according to Eifrem.
"The tension is with civil liberties, but in terms of efficiency, the more data you have, the better," he said.
The data collection is already well underway.
Security and intelligence agencies already collect a great deal of data, even if they might not yet be applying graph technology analysis to it.
National security agencies have added "socmint" -- or social media intelligence -- to the traditional ways of collecting intelligence: "humint," or human intelligence, the information collected by people; and "sigint," or signals intelligence, information pulled from electronic signals such as radar or weapons and communications systems.
US security agencies are now using and investing in social media monitoring tools like SocioSpyder, Dataminr, Geofeedia and PATHAR that scrape the web for information.
Civil liberties concerns
"The way the government is doing this amounts to surveillance," said Handeyside of the ACLU.
Neo Technology's software is already being used by US agencies. It's also been bought by a foreign equivalent of the US State Department that's trying to merge databases across its government to create a single view of its citizens.
"There tends to be a great deal of faith in the capacity of technology to ferret this out and basically predict the future," said the ACLU's Handeyside.
"But we're getting into very dangerous territory if we're talking about gobbling up more and more information about what people are saying, who they're communicating with, and where they're praying, and using that as the raw intel for making these kinds of judgments," he said.
There are some limits on government agency's access to data, of course. Beyond the publicly available streams of information on sites like Twitter and Facebook, law enforcement agencies like the FBI have to go through certain steps to gain any kind of access to a suspect's private accounts.
They're first required to send the suspect's email provider a National Security Letter asking, for example, for the suspect's search history. If that yields enough information to warrant further investigation, then they have to issue a search warrant to look more closely.
US intelligence agencies operate under different rules internationally, with more stringent limits on its approach to US citizens as opposed to non-citizens.
Clinton has said there's a need to tackle the "thorny challenge" of navigating security concerns and civil liberties, particularly surrounding encryption of mobile decides and communications.
She has expressed a preference for more targeted data collection over bulk data collection and supports the USA Freedom Act, the 2015 law that scaled back bulk collection of US citizens' telecommunications data by US intelligence agencies.
In the wake of 2013 disclosures by the NSA contractor Edward Snowden of the extent of surveillance, lawmakers believed creating the new law would restore Americans' trust.
Even so, distrust about the extent of collection and the reach of intelligence agencies persists. And even if a potential Clinton administration did succeed in moving forward with such an "intelligence surge," the graph technology is no panacea.
Lewis of CSIS said these new tools "probably could have predicted the attacks in Europe" that were larger, such as the Paris attacks or the March attack in Belgium. He and other security experts say that French intelligence agencies were hampered because they haven't updated their capabilities.
"The French didn't keep up, didn't invest in data analytics," Lewis said. "If you aren't looking at the data, you aren't going to find people."
But there are limits, he said.
"How do I create a program that's good at predicting who will be a lone wolf?" he asked. "It's harder to predict who's going to be a terrorist than it is to predict credit card fraud. We're not very good at that."