QlikView 11 looks great. Three new features in particular are going to make great impressions: the improved Web interface (AJAX), Session Sharing, and Notes.

No related posts.

“With enough data, you can discover patterns and facts using simple counting that you can’t discover in small data using sophisticated statistical and machine learning approaches.” Link

I used to assume that big data and data mining and statistics were inseparable. But the reality–companies making a killing transforming data into value–is far from complex.

Big data is not hard. Statistics are not required. Neither are complex algorithms. Google’s Marissa Mayer attributed the company’s intelligence to the volume of data available for cross-referencing and not to clever algorithms. Google translate leveraged massive volumes of cross-referenced text in multiple languages rather than a finely tuned understanding of grammar. Voice translation uses much the same technique based on huge volumes of recorded, transcribed text.

Right now our two best tools are visualization and data exploration (business discovery). Both are simple, easy to demonstrate and easy to grasp. The big data revolution’s message to the masses is that simple correlation will outstrip them both as long as enough data can be crunched. And much of this can be automated, pre-calculated, and even anticipated. Imagine the analysis system analyzing itself: these people tend to ask these questions at these times!

Data can be correlated post-hoc. Correlation does not equal causation, but simple correlation is ample evidence on which to take action. Correlation is immediately perceived visually. Correlation is relative and easy to compare. Correlation can look at 2, 3, 4 or more factors at once. Correlation is business friendly. It is easily understood. Correlation is gut-instinct compatible. Kids understand it: mom gets upset when I put peanut butter on the cat. If I do it right now, she’ll probably be mad.

The business opportunity is really that so much big data is simply thrown away. The opportunity to store all this data didn’t exist, so we have an old habit of simply letting it vaporize. Every server message, every website click, every customer contact and interaction, every manufacturing activity, temperature, timeclock action, phone call received, phone call placed, security video, email sent. Every bit of data can be analyzed, and from multiple perspectives: employee, employer, customer, vendor, shipper, receiver, and on and on.

We don’t know what we’ll find. As more and more stories of big data at little(er) companies emerge, the snowball will become an avalanche.

Related posts:

1. Low-Cost Data Analysis & Visualization: It’s Getting Better All The Time

A subtle and powerful shift happened last year.

I was building a QlikView application for financials. My client and I had discussed the idea of this application a year earlier but it was impossible then. Version 8.5 had not been released. The ability to look at dozens of simultaneous selection sets is the key to making this great idea work.

Zoom forward to 2009. Versions 8.5 and then 9.0 are released with features including Set Analysis, unlimited rows, chaining of document selections, data export from the script, and Dynamic Tables. These innovations remove the architectural limitations of QlikView that had tied my hands. A year ago I could not deliver the solution that was in my head and that my client needed. Now these limits are gone and I can build exactly what my client needs.

Build exactly what my client needs? This is the first time that this thought crossed my mind. It’s true! With the release of version 9, QlikView has entered a new phase. One that no other product can match.

QlikView is the first and ONLY tool on the market in which every business analysis that I have been asked to build can be built with confidence and an expectation of success. Dream big!

QlikView is not SPSS or JMP, and it never will be, but since I have never been asked to do anything more complex than a regression, QlikView works perfectly.

QlikView is the tool to turn to. It delivers results. Real value, right now. And you can be confident that it can achieve any business analysis you can think of. To get an idea of what QlikView can do, follow this thread on LinkedIn with over 100 unique uses for QlikView.

Related posts:

1. QlikView 8.5 Introduces “Set Analysis”
2. QlikView 9 Beta!!!
3. Seeing Dollar Signs in QlikView 8.5

QlikView 9 adds a powerful new feature to Set Analysis. In 8.5, you must specify the values to be selected in a field. But how can we dynamically make selections based on another field? QlikView 9 introduces “Set Modifiers with Implicit Field Value Definitions” in section 19.4 of the reference manual. Here’s an example:

sum( {$ Customer = P({1<Product={‘Shoe’}>} Customer)>} Sales )

The expression inside P(..) is saying “Start with the complete data set and select the value “Shoe” in the Product field. The list of customers after making that selection is what I want to use to calculate sum(Sales).” Ok, that’s great. It’s dynamically generated based on selections in another field. Now we don’t need a crazy concat() expression to acheive this like we did in 8.5. And by using E(..) instead of P(..), it’s now possible for the first time to select the excluded values! That wasn’t even possible in 8.5!

But the real power is the second Customer. You do not need to return a list of values from Customer in order to make a selection on Customer.

So what? My favorite application of this new ability is in using multiple calendar tables. Let’s say you want to look at sales between two different calendar periods…

• In the pre-8.5 method, without set analysis, you were limited to sum-if logic. In order to pick two dates and calculate over two periods, both calendars needed to be unconnected from the data model. Because these tables are unconnected, other list boxes would not update as the user made date selections.
• The 8.5 approach to multi-time-period analysis has been to use bookmarks and macros. You make one period selection and bookmark it, then make the next selection. One calculation applies the bookmark, the other does not. List boxes for other dimensions continue to update as new selections are made. The user cannot see the bookmarked date selections again without applying the bookmark. The bookmark can also contain additional, unintended selections. The way to avoid this is with macro code that makes selections, which may or may not kill the server selections cache, depending on who you ask.
• THE NEW  QlikView 9 way is to have two calendar tables in the data model, one connected and one unconnected. Using the new implicit selections feature we select the Possible dates in the unconnected table like so: {$Date=P({$} UnconnectedDate)>}. On the unconnected table, the user may have made selections on Year or Month or Day of Week, but we don’t care. We will select whatever UnconnectedDate values remain. By displaying both date tables we maintain visibility for both sets of date selections. Because one date table is connected, all the other fields continue to update. It is best therefore to use the connected date table as the current period and the unconnected table as the past period.

Hope you got something out of this. Cheers!

Related posts:

1. Missing Feature In Set Analysis
2. QlikView 8.5 Introduces “Set Analysis”
3. The Year in QlikView

An example of their interactive analysis tool. The data is from Amazon Web Services but I’m pretty sure it is not connected “live” in any way. Last I checked, Spotfire loads data as a batch process.

Related posts:

1. Response to the Tableau 3.0 Webinar

Peter Batty discusses one-second results in the world of geospatial data.

The first was that if you can provide information at “the speed of thought”, or the speed of a click, this enables people to do interesting things, and work in a different and much more productive way. Google Search is an example – you can ask a question, and you get an answer immediately. The answer may or may not be what you were looking for, but if it isn’t you can ask a different question. And if you do get a useful answer, it may trigger you to ask additional questions to gain further insight on the question you are investigating.

A second idea is that when you are looking for insights from business data, the most valuable data is “on the edges” – one or two standard deviations away from the mean. This leads to another Netezza philosophy which is that you should have all of your data available and online, all of the time. This is in contrast to the approach which is often taken when you have very large data volumes, where you may work on aggregated data, and/or not keep a lot of historical data, to keep performance at reasonable levels (historical data may be archived offline). In this case of course you may lose the details of the most interesting / valuable data.

Related posts:

1. How One-Second Results Change Everything

There’s a point where query response time is low enough that it changes the analysis game completely. This is the amount of time that a decision maker is willing to wait to get the next answer. Not the first answer, but the next one, and the next one. Eventually the frustration of waiting is worse than not knowing.

Salesperson: “What shipped yesterday? Ok, what’s the breakdown? Woah, what happened in that department? That markdown is too steep, who wrote that order? Which customer? What’s that rep’s extension?”

With one-second results, that analysis would have happened in the time it took you to read it. This is a competition against human nature. One-second results makes the difference between wishing you had the answer and getting it, multiplied over and over throughout the day.

The impact on a business is not from faster queries alone. Behavior changes when decision makers trust that the data is immediately at hand. The relationship to data changes when you can find the answer while you think about it and not lose your train of thought.

Because the query engine can respond to any query in one second, we can make every path of exploration available at the beginning. One application can take the place of many reports. Users can begin to query immediately and along any drill path. The benefit of one-second results is diminished if users have to first identify the report that has the data and filtering options they need.

Can OLAP deliver this? No. We must combine speed of execution with rapid application development, full transaction details, and eliminate predefined drill paths. OLAP/MOLAP/ROLAP/SCHMOLAP can’t take us into this new era. In-memory associative and column-store databases can.

With one-second results, you don’t build a query and then start the execution. Instead, the results update as soon as you pick the first filtering option, whether it’s the day, order number or country of origin. You get immediate feedback before you make your next selection. Also, the filter options can change based on the results. Maybe you remove options that are incompatible with the selections made so far. By shrinking the feedback loop with one-second results, the filtering options can show intelligent behavior to help guide users or add context to the results. This level of dynamism lets users roll back and forth through their ideas. They can cross-reference without losing a train of thought, or discover and follow tangents that are more important.

It’s not just one decision maker getting an answer quickly. Interactions and processes benefit. Workers get feedback in near-real-time. We can do tricks like running the same query once per second. Ridiculous? This isn’t paradise, I live in the land of low budgets and “getting it done”. Vendor and customer data is available right when they’re on the phone. Less “I’ll get back to you” and more “I have that info right in front of me.” I’ve also noticed that it’s harder to bullshit when anyone in the meeting can easily explore the data on their laptop and get the real answer.

In companies where I can deliver one-second results, I spend a lot of time reconditioning people to ask for anything they desire, because now I can put any information at their fingertips, no matter how many tables, how much detail and with little knowledge of how they want to look at the data.

For nearly all companies, the entire transactional database can be copied as-is into a one-second query engine. Add a BI tool on top, rename some fields and identify the table relationships. Time is spent developing the frontend to deliver the best reports and analysis. One person can build the entire solution. Since the transactional model is already validated, there is no data modeling, no formal architecture and little documentation. This might be frightening to enterprises but the benefits are huge for strapped IT budgets.

A one-second query engine needs an interactive frontend to take advantage of it. We also need simpler ETL tools. With the engine in place first, developers will connect the dots and the tools will be built to take advantage of the new abilities.

None of this is theoretical. I’ve been doing this for the past 7 years with an in-memory associative database, ETL tool and interactive frontend called QlikView. When information flows at the speed of thought, it changes decision-maker behavior and the business process. When we can prototype and deploy one-second query engines quickly, then ideas can be built and tested quickly. Most ideas won’t be new or unexpected, but they were impossible or impractical without one-second results.

Related posts:

1. Peter Batty Discusses One-Second Results And Geospatial Analysis
2. Low-Cost Data Analysis & Visualization: It’s Getting Better All The Time
3. What makes QlikView so good?

Over the weekend I have revisited Tableau, enjoyed some success with MonetDB, tried to turn MySQL into a hundred million row data warehouse, been underwhelmed with Firebird, installed Greenplum and spent many frustrated hours with Talend Open Studio, Pentaho Kettle and Jitterbit.

Of course, I could just buy QlikView, but what can be done for less $money? Unfortunately data warehouses and BI front-ends are not sexy problems in the opensource community. Graphs and charts get a little more attention, but you’ll need to write your own code to glue them to your application.

In summary, what can I say about our options?

First, write your own ETL. Why do opensource ETL tools like Talend and Kettle work so hard to rebuild Informatica? It reminds me of Linux in the 1990s when the community wanted to beat Windows and kept working to look like Windows and wondering when victory would arrive. Informatica, like OLAP and mainframes, is from an era when memory was scarce; languages were low-level, slow to compile & run, abstracted little and were not at all portable. On top of that, ODBC drivers were tightly controlled and costly.

But now we can pick from many great scripting languages. Today’s languages abstract the hard parts, are easy to read, can be edited while executing and talk to any system, database, web service or application. I think the next direction for ETL will be a simple (but extensible) transformation language using an ORM wrapper… Rails on ETL. Until that arrives, you can achieve everything you need with PHP, Perl, Ruby and others.

Best option for low-cost data warehouse?

Check out the totally free MonetDB. Unless Vertica or InfoBright reconsiders releasing a low/no cost option, MonetDB will likely mature to become a first-choice column-store database. It’s an academic project that has earned a sizeable development community and user base. The product is functional today for tens of millions of rows (maybe more). So far I have personally worked with a few million rows in MonetDB and I’d like to use it again. With a little focus on usability and packaging, it could be a contender.

Greenplum, freely available for development, won’t help. The architecture is designed around Massively Parallel Processing. As a single, standalone installation, it’s basically just PostgreSQL. You won’t see extra performance without a farm of servers.

To my surprise, MySQL itself is not too bad. The MyISAM tables are speedy and Alex Tomic wrote a post about using multiple queries against the Archive storage engine and how to steal an index with that engine. With basic MyISAM on a fast server, I’m running 10GB table scans in under a minute, but moderate aggregations take a few minutes. Architecturally, MySQL is limited. One query = one thread = one core. Running two simultaneous queries is an option, but MySQL still would not do the kind of transparent, optimized caching that you need for a warehouse. Throughput is limited to disk I/O speed. InfoBright has built a column-store storage engine for MySQL but it’s targeted for the enterprise only.

What about the front end?

For the money and quality and ease of integration, it’s hard to beat Tableau. $1800 bucks isn’t cheap, but for a small business that truly needs to analyze patterns, this will do the job and it makes very pretty charts. The most recent version has integrated support for mapping based on zip code, area code, state, country and others. The maps also incorporate Census and USGS data and are pulled live from an online source. They look great! Tableau has always had a smooth, easy-to-understand layout and a crisp look that makes each chart very attractive in a presentation. It also automatically guesses what chart you want based on the quality & number of aggregates and dimensions.

The drawback is that Tableau doesn’t have its own high-speed database or ETL tool. Tableau can’t shine until a low/no-cost read-optimized database is available. Until then, it does support the most common databases and data warehouses, both commercial and open-source. Except it can’t handle generic ODBC and I don’t know why.

There’s JasperSoft = CrystalReports + OLAP + Informatica + Web Dashboards. Each component is from a different opensource project, so they don’t all use the same platform or interface, and they can’t all read the same data sources. The democratization of BI is NOT going to come from enterprise tools made cheap; it will come from simple disruptive tools that add new ideas and polish with each release. Sorry, Jasper.

What would I use to build a reporting system for a smaller business?

Well, assuming we’re doing it to make more money, not to keep up appearances, the best choice is still to pay the money for QlikView. It reads ODBC, OLE DB, text files and Excel–everything a business needs. The ETL language is easy to understand for any businessperson that has put together an Access database or enjoys Excel formulas (blech!). The GUI front-end designer is powerful & straightforward. And the in-memory database behind QlikView is so incredibly fast that I routinely analyze 10 million of rows in a split-second. It’s a one-stop shop.

Tableau is a good option but you lose the database and ETL. Maybe you don’t have a large volume of data or maybe it’s all in one view in the database–Tableau could work for you.

At a lower cost? Well, it definitely comes down to tradeoffs in coder skill, money, development time and ease of use. Whereas in QlikView anyone can write the basic code to read a couple tables, all other solutions demand heavy lifting somehwere.

If I was doing it for free?

I’d start with PHP, and possibly Ruby. Read from a database, calculate, generate Google Charts, and maybe use one of the low/no-cost Flash-based charting libraries for interactive splash. In a future post I’d like to cover ORMs and Google Chart APIs and how it can help get these projects off and running quickly.

Got any ideas? I’m always on the lookout for a faster cheaper better way to create these solutions.

50 Cool Things You Can Do with Google Charts

Related posts:

1. Interactive Information Visualization
2. U.S. Army Tests Real-Time 3D Visualization Of On-The-Move Data
3. How One-Second Results Change Everything

I consult for QlikView and I have to agree it’s awesome. But hearing me rant about its greatness would sound like another fanboy foaming at the mouth. So I’ll let someone else, David Raab, explain why QlikView is so good. David has also put together a concrete example using a cross-sell table that answers the question “What other products do customers tend to buy if/when they purchase product X?” This is a powerful question that every sales person should be asking, but it’s hard to get an answer when you need to go to the IT department each time and wait for them to build the model in a traditional SQL query or OLAP tool. QlikView makes it easy to get immediate answers and explore your data “at the speed of thought”.

Related posts:

1. Peter Batty Discusses One-Second Results And Geospatial Analysis
2. Interactive Information Visualization
3. How One-Second Results Change Everything

What if you could turn on a massively parallel business intelligence database cluster with a few lines of code? What if you could leverage in-house and outsourced resources for computation and storage as needed? What if you could expand your analysis, data mining and text-search effort one node at a time, transparently, instantly?

There’s been a flurry of discussion around Hadoop and the Hbase project to bring Google’s BigTable feature to Hadoop.

Now Amazon wants to talk about how to use Hadoop with EC2 and S3, their computing and storage clusters.

Can I search large volumes of data on the cheap? Yes, but my algorithms must fit within the MapReduce framework.

Does someone have a MapReduce-enabled data query language? Well, there’s Pig from Yahoo. Sawzall from Google. Here is a discussion comparing those two from Greg Linden. Abacus from the Hadoop project. Apparently Microsoft has DryadLINQ.

We are on the exponential curve as it swoops upward dramatically. From the power and flexibility of opensource, anyone can use Google secret sauce on Amazon’s computers for 18 cents per gigabyte and 10 cents per computing hour.

Related posts:

1. Low-Cost Data Analysis & Visualization: It’s Getting Better All The Time
2. Data Warehousing In Education
3. InfoBright Open-Source Column-Store DBMS