Big Metadata is my phrase for the application of Big Data tools and techniques to the larger ecosystem of data surrounding business software. While Big Data is focused on unwieldy data sets at large firms or new-technology companies, Big Metadata is about every company or organization taking advantage of the massive amounts of relevant metadata that is completely ignored.
Business software only capture the bits of each transaction that will calculate changes in such things as inventory, bank accounts, and production. This reductionist view of each transaction is disconnected with the reality of daily business: full of exceptions and rewrites and human conversations. Just as consumer software is making strides in conforming to our habits and needs, business software should capture the reality of business and not enforce an ideal.
How we got here is simple. The constraints on storage, processing, software development and error checking left us focused only on critical features. We now design business databases to be as narrow as possible. We write code to run the system for today and struggle to reconcile original decisions as processes, people and the whole business landscape continues to change.
When we talk of business software being more human, more predictive, and more like the consumer software we experience from Apple and others, I think we are looking for the same kind of Big Data solutions that come from Google, Apple, Amazon, and Facebook. These are solutions based on massive amounts of data, good choice in analysis factors, and a sprinkling of algorithm and statistics.
There are clear individual business uses for social networks, voice and gesture recognition, cloud infrastructure, schema-less databases, in-memory storage, distributed algorithms, graph processing, domain-specific languages, development frameworks and other technologies. I believe the combination of these technologies to solve the common challenges in business will fuel the next great wave. My hope is to sketch out the future of business software.