Tag Archives: business

Delusion

Behind the Romney Victory Delusion

We don’t know why the internal numbers of the Romney campaign were so far off, but I don’t think there’s an excuse. The fact is that Nate Silver, Princeton and others used models that once again predicted states correctly (Nate scored 100%), plus Senate and House races too. Reality was not only available, it was free.

This is one of the human factors of analytics: reality is not always welcome. You can beg that management not shoot the messenger, but practice ducking anyway.

1 Million Lessons For Netflix About Big Data

Netflix PrizeNetflix opted not to implement the entire algorithm that won the Netflix Prize because a) it would have cost too much to engineer, b) it no longer fit the direction of the company, and c) just two algorithms from the prize submission yielded the most significant gains.

This is such a great example of the reality of Big Data! Before embarking on a mathematical journey, companies need to assess the reward of developing models and balance against the risk of process changes, and the cost of implementation. Also, simpler models that are cheaper to implement and better understood in the general case are more likely to get business traction than deep models, highly tuned through iteration.

Netflix’s factors changed, meaning the pieces of data fed into the system. Factor selection is repeatedly identified as the most significant step in model building. With streaming, a partial viewing of a video means something: so do clicks to go back, forward, or browse for something else while you’re watching. All of this reflects your behavior.

Big Metadata: The Storm of Business Data All Around Us

Big Metadata is my phrase for the application of Big Data tools and techniques to the larger ecosystem of data surrounding business software. While Big Data is focused on unwieldy data sets at large firms or new-technology companies, Big Metadata is about every company or organization taking advantage of the massive amounts of relevant metadata that is completely ignored.

Business software only capture the bits of each transaction that will calculate changes in such things as inventory, bank accounts, and production. This reductionist view of each transaction is disconnected with the reality of daily business: full of exceptions and rewrites and human conversations. Just as consumer software is making strides in conforming to our habits and needs, business software should capture the reality of business and not enforce an ideal.

How we got here is simple. The constraints on storage, processing, software development and error checking left us focused only on critical features. We now design business databases to be as narrow as possible. We write code to run the system for today and struggle to reconcile original decisions as processes, people and the whole business landscape continues to change.

When we talk of business software being more human, more predictive, and more like the consumer software we experience from Apple and others, I think we are looking for the same kind of Big Data solutions that come from Google, Apple, Amazon, and Facebook. These are solutions based on massive amounts of data, good choice in analysis factors, and a sprinkling of algorithm and statistics.

There are clear individual business uses for social networks, voice and gesture recognition, cloud infrastructure, schema-less databases, in-memory storage, distributed algorithms, graph processing, domain-specific languages, development frameworks and other technologies. I believe the combination of these technologies to solve the common challenges in business will fuel the next great wave. My hope is to sketch out the future of business software.