Within the vast, undifferentiated torrent of data that courses through the Internet, there hides an intricate topology of information. Decision makers with millions of dollars on the line need a much more detailed map of that information texture than mass-market engines like Google can provide. For bespoke, targeted data curation, corporations and governments turn to a young company called Quid to find hidden connections in the maelstrom of data that can make fortunes or save lives.
Founded by Yelp! pioneer Bob Goodson and Oxford-trained physicist Sean Gourley, Quid uses a collection of proprietary algorithms to trawl the Internet for all available linguistic and quantitative data, map the connections running through all that data, and provide a history of how those connections evolve. To Quid’s clients, the constellations of data revealed by those algorithms make actionable sense out of otherwise incomprehensibly complex systems such as global industry, scientific innovations, and even violent insurgencies.
“The world is more complex than the human mind can get around, but that doesn’t stop the need for decisions to be made,” Gourley says. “If you have a tool that helps people make better decisions in this complex world, that’s worth a lot of money.”
Quid’s process works on three levels.
First, a custom search algorithm trawls the web for linguistic and quantitative data. Quid then builds a web of connections from those source keywords, numeric content and citations. For instance, one data tapestry might link the cost of electronics on retail sites to news reports of political instability in a developing nation, because a research paper mentioned that a key component of some gadget requires a mineral mined in the unstable country.
“Kind of like Google, we suck the information in. New York Times, Twitter, academic journals, job searches, every scientific journal that comes out, you name it, it all goes into the algorithm,” Gourley said. “Machines can see those connections very well. That’s kind of the point. Things are connected in ways we never thought, or didn’t have the power to think of.”
Next, a customer approaches Quid with questions about a specific research area or investment space. A different algorithm filters the vast map down to only the connections relevant to that inquiry. Finally, Quid provides a visual representation of that allows the customer to clearly understand the present state and recent past of the area in question. The visualization takes the form of connected clusters of 50 to 70 nodes, into each of which users can drill down, revealing more nodes and deeper complexity.
To customers interested in a short time horizon, up to 18 months or so, the clusters of nodes provide the important information. Those nodes are where the action is now, areas where companies should strike while the iron’s still hot. To customers looking out further, five to ten years out, it’s the white space between the nodes, empty spots that represent untapped opportunities for growth or new areas for interdisciplinary research, that communicate the most useful information.
What Quid does not do is predict what happens next.
“It was never about predictions. Everyone wanted it to be about predictions, but we want to just understand what’s going on today. And answering that question is hugely valuable. Forget about tomorrow, people can’t even understand what’s going on today,” Gourley said. “Now, of course, the second step is asking ‘where were they yesterday?’ Then you get a time series, and you have the choice to decide if you think things will continue on tomorrow as they behaved before.”
The business has already proven profitable for Quid, drawing in both a steady stream of customers and over $10 million in funding from various investors. But moving forward, it’s not the just the money that intrigues Gourley. As he sees it, Quid is as much a vehicle for his ongoing research about uncovering the interesting hidden mechanisms behind the world’s complex systems as it is a consulting service well-funded organizations.
“For me, the desire has always been to understand these global systems that dictate and govern the world we live in. The price you pay for that is you have to be able to generate revenue,” Gourley said.
“But any time you solve a hard problem, you will generate revenue. There’s a nice balance between the scientific endeavor and the commerce.”
The post Video: A Company’s Algorithms Reveal Hidden Connections Among All That Data appeared first on Popular Science.