Druid is an in-memory high-speed analytic database that is distributed in a cloud platform. MetaMarkets says they plan to release the code in the coming year. Data is fed in a semi-aggregated form from a Hadoop backend that stores the raw data. Their cluster of 40 modest machines churns through 1 billion rows of arbitrary dimensionality in 1 second. In another post they mention that they can use 6TB of memory spread among multiple machines before they incur degradation in speed due to cloud communication overhead.

Related posts:

1. InfoBright Open-Source Column-Store DBMS
2. QlikView in the Cloud
3. Pentaho Open-Source BI Suite Adds Weka Data-Mining Tool

“With enough data, you can discover patterns and facts using simple counting that you can’t discover in small data using sophisticated statistical and machine learning approaches.” Link

I used to assume that big data and data mining and statistics were inseparable. But the reality–companies making a killing transforming data into value–is far from complex.

Big data is not hard. Statistics are not required. Neither are complex algorithms. Google’s Marissa Mayer attributed the company’s intelligence to the volume of data available for cross-referencing and not to clever algorithms. Google translate leveraged massive volumes of cross-referenced text in multiple languages rather than a finely tuned understanding of grammar. Voice translation uses much the same technique based on huge volumes of recorded, transcribed text.

Right now our two best tools are visualization and data exploration (business discovery). Both are simple, easy to demonstrate and easy to grasp. The big data revolution’s message to the masses is that simple correlation will outstrip them both as long as enough data can be crunched. And much of this can be automated, pre-calculated, and even anticipated. Imagine the analysis system analyzing itself: these people tend to ask these questions at these times!

Data can be correlated post-hoc. Correlation does not equal causation, but simple correlation is ample evidence on which to take action. Correlation is immediately perceived visually. Correlation is relative and easy to compare. Correlation can look at 2, 3, 4 or more factors at once. Correlation is business friendly. It is easily understood. Correlation is gut-instinct compatible. Kids understand it: mom gets upset when I put peanut butter on the cat. If I do it right now, she’ll probably be mad.

The business opportunity is really that so much big data is simply thrown away. The opportunity to store all this data didn’t exist, so we have an old habit of simply letting it vaporize. Every server message, every website click, every customer contact and interaction, every manufacturing activity, temperature, timeclock action, phone call received, phone call placed, security video, email sent. Every bit of data can be analyzed, and from multiple perspectives: employee, employer, customer, vendor, shipper, receiver, and on and on.

We don’t know what we’ll find. As more and more stories of big data at little(er) companies emerge, the snowball will become an avalanche.

Related posts:

1. Low-Cost Data Analysis & Visualization: It’s Getting Better All The Time

QlikView’s “section access” security can be easily ignored with a hacked copy of QV.EXE. Hacking the file is a simple process that takes only a few minutes. Do not rely on “section access” to protect your data in a local QVW file.

The effectiveness of “section access” on server-hosted files is not a part of this warning.

Related posts:

1. Using QlikView Tickets

The positions have been filled so I have removed this post.

Related posts:

1. Carlyle Group To Build H&M Distribution Center In Hamburg
2. Outstanding Article On How To (Really) Build A Business Data Warehouse
3. Microsoft Photosynth

Besides my love of tech, I have an unusual fascination with international trade, container ships, logistics and other big global systems that quietly hum along day and night. There’s a very nice render of a shipping container on my business cards.

I’m thrilled to stumble upon the blog of Ethan Zuckerman and his mention of a new BBC series “The Box” that is tracking a cargo container around the world for a year.

What is the current location and load of the box?

UPDATE: That link really is the current location. I was expecting to be disappointed, but since I first saw the page, it has indeed been migrating across the map toward its very first load, of whiskey.

No related posts.

Harry Debes says “SaaS” will die out in 2 years.

Vinnie Mirchandani, Joshua Greenbaum, Bob Warfield respond.

I certainly agree with Mr. Debes that we have been here before. Technology analysts grab hold of the next great thing and see how far it will bend before it breaks. SaaS is a collection of ideas exemplified by Salesforce. Each idea could be implemented by traditional software vendors. We want to understand what makes Salesforce successful, but we must recognize the difference between Salesforce and an ERP system. Sales pipelines can be abstracted from the daily operations of a business. Salesforce does not need to understand backflushing with 3-level bills of material with multiple currencies and the impact on materials planning in a cellular manufactring environment. Thankfully.

Mr. Mirchandani says the problem is that enterprise software vendors are not listening to their customers who are asking for Salesforce features and a Salesforce pricing model. Perhaps for some that makes sense, but what makes Salesforce and other SaaS companies successful is the tight scope of their product. Enterprise software has a huge scope and it must be completely customized in certain areas. Areas that are different for every type of business.

Related posts:

1. Salesforce.com’s Force and SAP’s A1S
2. More About Salesforce.com’s AppExchange 2.0 Offerings
3. What Software Was Hans Rosling Using?

We design and build solutions that increase productivity in your business!

What kind of productivity?

Business productivity!

I guess you do it all. I’d like that.

More productivity?

I want more profit.

We can help with that!

You can? Why don’t you just say, “We make you more money.”

We improve the productivity of your business!

That’s not the same thing at all. “Productivity” is vague and unmeasurable.

We can help!

Can you help me make more money?

We can help you become more nimble!

What does that even mean? Can anyone measure it?

We help you compete in a global marketplace!

I’m a lot more concerned about the competition knocking on my customer’s door today.

We can help you manage your customer relationships!

Will that make me more money?

It helps productivity!

So you want to sell me services to improve my productivity, but they might not make any money?

We can do whatever you want us to do!

Your website says you have the “skills and depth of experience” to help my business be more productive.

You want to be more productive?

I’d like to make more money.

We can help!

Related posts:

1. Microsoft Demonstrates Industry-Specific Office Collaboration Solution

It figures that I’d be upset about the lack of quality in the ETech presentations and walk right into an excellent talk by Nathan Eagle of MIT. Nathan’s research included giving phones to students and faculty at MIT that tracked location (both as a cell tower and as described by SMS), proximity to other Bluetooth-enabled phones, and the state of the phone such as talking or charging. Then the audience was floored by the ease of predicting behavior, inferring relationships, and determining personal attributes of the user.

No related posts.

Lawrence Lessig speaks tonight in a keynote that I’m certainly looking forward to. Generally, ETech includes too few presentations that are delivered by alpha geeks and concerned with emerging technology. If this really is the next wave of innovation being presented here, what’s missing is why and how it will make a difference. A solid presentation about emerging technology needs to outline what the questions are that have yet to be answered, the possible future, the impact and things like that. All these qualities were included in Stan William’s talk, Saul Griffith’s talk and today’s talk from Mike Walsh about the “Asian Media Revolution”.

Related posts:

1. ETech 2008
2. Data Warehousing In Education

I left my iPhone in a cab on Monday coming in from the airport. Bummer. On the bright side, I talked to a rep from Jaduka today. They have a web demo of their tools that I can use to make free calls from my hotel room. There are other options, for a variety of reasons this one works perfectly for me right now.

No related posts.