With 90 percent of the world’s data created in the last two years, what can we expect our data vaults to hold two or even twenty years from now? Today we measure our lives in peta-bytes, but by 2020 estimates show a 2,300 percent increase in the bits and bytes that will define our lives. 35 zeta-bytes to be exact. How then can we as a society leverage the intrinsic value of so much data without getting bogged down with its complexity?
Around the turn of the century, we experienced a similar moment of euphoria when retail outlets opened "virtual stores" and sold products to online buyers. A famous IBM TV ad once depicted an overwhelmed young company whose products went from a few online orders a day to hundreds of thousands. In many respects we have come full circle and are back at the starting gate of yet another era of unprecedented growth only this time instead of millions of orders, the focus is on zillions of data points.
In 2000 CEOs focused primarily on IT integration and supply chain strategies to fulfill a surge of orders. Their managers implemented the latest e-commerce packages, leveraged the cloud to reduce costs, broadened and compressed their global supply chains, and trained their workforce to adapt new work flows. Success was determined from a customer's positive experience, measured primarily by the number of accurate and timely deliveries.
Today, the paradigm has shifted away from a transaction-centric one to customer-centric. Companies no longer wait for customers to buy but instead develop sophisticated algorithms that can compare a specific customer’s purchase history with multiple data sets including credit rating reports, recent purchases, and most extraordinarily, their genuine propensity to buy based upon the web pages they most commonly visit. Surprisingly, web behavioral data has become a powerful data complement that can offer unprecedented efficiency benefits to both the merchant and the consumer. Customers receive compelling suggestions, while stores inventory the products their customers will most likely purchase. It's a win-win for both. Issues of privacy remain a sticking point for some individuals, but, as the benefits to the consumer improve, even these issues are expected to become less significant.
Striking the optimal balance will be tricky especially when the journey also involves flogging through mounds of unstructured web data. One approach being talked up within academic circles is systems thinking.