The futurist: Will big data destroy the stock market?

(Editor’s note: This is the first of two parts.)

When you buy a stock, you place a bet on how that stock will perform in the future. In a perfect world, where market insiders and manipulators are removed from the equation, the market is a terrific tool for determining the true value of companies being invested in.

But what happens when the volume of data used to make decisions increases 100 million times, and trading volumes increase 100 million times, and trades can be transacted over 100 million times a second?

The proliferation of mobile phones, social media, machine data, and web logs has led to massive amounts of data being generated, stored and processed, and this volume is increasing exponentially with the digital shift from offline to online. Inside these rapidly expanding data pools are millions of tiny little “tells” that can be extracted and combined with the emerging science of anticipatory computing into very predictable movement indicators.

At the same time, these “tells” can be fabricated, manipulated, and auto-generated. In less time than it takes for a human to blink an eye, the entire value of the markets can fluctuate over $100 trillion and back again without leaving any humanly understandable trace of what just happened.

How long will it be before a series of trading algorithms go terribly wrong, causing the markets to simply implode? Here are a couple thoughts.

Moving into the Big Data Era

First a few jaw-dropping stats on the growth of structured and unstructured data. Here are a few examples:

• RFID (radio frequency ID) systems generate up to 1,000 times the data of conventional bar code systems.
• Smart dust products and similar “Internet of Things” devices connected to the Internet will reach 50 billion by 2020, each with its own 24/7 data generating capability.
• 10,000 payment card transactions are made every second around the world.
• Wal-Mart averages more than 1 million customer transactions every hour.
• 340 million tweets are sent per day. That’s nearly 4,000 tweets per second.
• More than 5 billion people are calling, texting, tweeting and browsing websites on mobile phones.
• 247 billion emails are sent every day (80% is spam!)
• It now only costs $600 to buy a disk drive large enough to store all of the world’s music.
• 30 billion pieces of content are shared on Facebook every month.
• Projected growth in global data generated annually is 40%. By 2020, the production of data will be 44 times greater than what we produced in 2009

Big Data is indeed …big! And getting bigger.

The Future Stock Market

When it comes to the stock market, humans are already the outsiders. The number of actual trades transacted by real humans is only a small fraction of a day’s total trading volume.

Yes, there are still people who buy stocks because they “like” the company, and yes, companies who are attempting to recruit key employees will still give stock options as part of their compensation package.

But for those who are serious traders, buying and selling stock on a daily basis, the game is constantly amping up to new levels where automated systems do most of the grunt work.

In a trash-talking society with quants boasting “my algorithm can beat your algorithm,” the entire stock exchange has deteriorated into a kind of bot vs. bot warfare that has lost all semblance of being a marketplace of human values.

One Possible Scenario

As a way to help you understand how this may unfold, here is a short scenario.

With the stock market, every trade creates a butterfly effect, a tiny ripple of influence, with the size of the ripple being directly proportional to the size of the trade. When trades happen at lightening speed, the size of the ripples can be manipulated through a series of external distortion signals designed to mislead all of the other bots, leading to ever-greater buy-sell patterns until they reach a point of maximum distortion for profit taking.

If, as example, a quant were to set up a specific trading pattern starting with 2 shares, followed over a set period of time by a series of doublings, 4 shares, then 8, 16, 32, 64, 128, 256, 512, and so on, the exponentially growing signal produces a “resonancing effect.”

Whenever a pattern like this is repeated multiple times, other bots will pick up on it, and start formulating strategies around it. For the puppet masters this becomes a relatively easy way to manipulate the market by first establishing a pattern, and then skipping or altering the final trade.

Much like the cracking of a whip, where all of the attention is focused on the final snap, the market positions itself for the grand finale. Predictability creates manipulability. It’s games like these that polymaths live for.