... and points beyond » Visualization

Great Features In QlikView 11

Jay Jakosky — Thu, 20 Oct 2011 23:22:47 +0000

QlikView 11 looks great. Three new features in particular are going to make great impressions: the improved Web interface (AJAX), Session Sharing, and Notes.

No related posts.

The Myth and Mystery of Big Data

Jay Jakosky — Wed, 12 Oct 2011 05:27:01 +0000

“With enough data, you can discover patterns and facts using simple counting that you can’t discover in small data using sophisticated statistical and machine learning approaches.” Link

I used to assume that big data and data mining and statistics were inseparable. But the reality–companies making a killing transforming data into value–is far from complex.

Big data is not hard. Statistics are not required. Neither are complex algorithms. Google’s Marissa Mayer attributed the company’s intelligence to the volume of data available for cross-referencing and not to clever algorithms. Google translate leveraged massive volumes of cross-referenced text in multiple languages rather than a finely tuned understanding of grammar. Voice translation uses much the same technique based on huge volumes of recorded, transcribed text.

Right now our two best tools are visualization and data exploration (business discovery). Both are simple, easy to demonstrate and easy to grasp. The big data revolution’s message to the masses is that simple correlation will outstrip them both as long as enough data can be crunched. And much of this can be automated, pre-calculated, and even anticipated. Imagine the analysis system analyzing itself: these people tend to ask these questions at these times!

Data can be correlated post-hoc. Correlation does not equal causation, but simple correlation is ample evidence on which to take action. Correlation is immediately perceived visually. Correlation is relative and easy to compare. Correlation can look at 2, 3, 4 or more factors at once. Correlation is business friendly. It is easily understood. Correlation is gut-instinct compatible. Kids understand it: mom gets upset when I put peanut butter on the cat. If I do it right now, she’ll probably be mad.

The business opportunity is really that so much big data is simply thrown away. The opportunity to store all this data didn’t exist, so we have an old habit of simply letting it vaporize. Every server message, every website click, every customer contact and interaction, every manufacturing activity, temperature, timeclock action, phone call received, phone call placed, security video, email sent. Every bit of data can be analyzed, and from multiple perspectives: employee, employer, customer, vendor, shipper, receiver, and on and on.

We don’t know what we’ll find. As more and more stories of big data at little(er) companies emerge, the snowball will become an avalanche.

Low-Cost Data Analysis & Visualization: It’s Getting Better All The Time

Can a humble Chart object get some love?

Jay Jakosky — Fri, 01 Apr 2011 15:09:03 +0000

If there was just one question I could ask at this year’s Qonnections 2011, it would be this…

When are we going to see improvements to the most basic QlikView task: displaying data?

Look at the following examples from competitors…

Above is a Spotfire chart that cleanly displays a 2-level hierarchy of dimension values on the x-axis. Increase to 3 levels and the labels stay organized and readable.

Below is a chart from Tableau.

The axis labels are only shown at the left and the bottom of the entire trellis. QlikView shows axis labels on each square, adding unnecessary clutter that is not easy to remove. Two dimension values are coded in the size of the dots and their color. Tableau also uses color gradients easily and effectively.

Tableau and Spotfire put a lot of energy into making displays clean and readable. Tableau makes excellent guesses at how to display your data.

QlikView’s charts have felt clunky for years. The Chart building dialog is huge, confusing and too often doesn’t work as expected. Charts don’t adapt well to being small. Axis labels cram into each other, don’t split lines, and don’t respect chart settings. Legends use excessive real estate, have limited positioning with no intelligence and don’t split text. Expression cycles are confusing for end-users. Fonts and colors are buried 3 levels deep. “Themes” exclude certain chart elements, requiring the developer to dive deep into menus to make targeted changes. Scatter plots quickly become a messy jumble of points and labels. Removing scatter plot data point labels makes identifying a data point a painful task of color matching.

There doesn’t seem to be any point in discussing geospatial data, for which QlikView has no native abilities. QlikTech has been frustratingly quiet on this. Want to include Google Maps? You’re welcome to search for code in the community, or pay more for third-party tools. Meanwhile, the competitors’ native support is easy and attractive.

QlikView is still the best tool out there for “getting things done”. Graphical display is one of a few areas where QlikView is lagging. But QlikView is too far behind at this point. Charts have not been overhauled since at most version 7. It’s time for a major leap forward.

Low-Cost Data Analysis & Visualization: It’s Getting Better All The Time

Jay Jakosky — Mon, 08 Sep 2008 02:43:45 +0000

Over the weekend I have revisited Tableau, enjoyed some success with MonetDB, tried to turn MySQL into a hundred million row data warehouse, been underwhelmed with Firebird, installed Greenplum and spent many frustrated hours with Talend Open Studio, Pentaho Kettle and Jitterbit.

Of course, I could just buy QlikView, but what can be done for less $money? Unfortunately data warehouses and BI front-ends are not sexy problems in the opensource community. Graphs and charts get a little more attention, but you’ll need to write your own code to glue them to your application.

In summary, what can I say about our options?

First, write your own ETL. Why do opensource ETL tools like Talend and Kettle work so hard to rebuild Informatica? It reminds me of Linux in the 1990s when the community wanted to beat Windows and kept working to look like Windows and wondering when victory would arrive. Informatica, like OLAP and mainframes, is from an era when memory was scarce; languages were low-level, slow to compile & run, abstracted little and were not at all portable. On top of that, ODBC drivers were tightly controlled and costly.

But now we can pick from many great scripting languages. Today’s languages abstract the hard parts, are easy to read, can be edited while executing and talk to any system, database, web service or application. I think the next direction for ETL will be a simple (but extensible) transformation language using an ORM wrapper… Rails on ETL. Until that arrives, you can achieve everything you need with PHP, Perl, Ruby and others.

Best option for low-cost data warehouse?

Check out the totally free MonetDB. Unless Vertica or InfoBright reconsiders releasing a low/no cost option, MonetDB will likely mature to become a first-choice column-store database. It’s an academic project that has earned a sizeable development community and user base. The product is functional today for tens of millions of rows (maybe more). So far I have personally worked with a few million rows in MonetDB and I’d like to use it again. With a little focus on usability and packaging, it could be a contender.

Greenplum, freely available for development, won’t help. The architecture is designed around Massively Parallel Processing. As a single, standalone installation, it’s basically just PostgreSQL. You won’t see extra performance without a farm of servers.

To my surprise, MySQL itself is not too bad. The MyISAM tables are speedy and Alex Tomic wrote a post about using multiple queries against the Archive storage engine and how to steal an index with that engine. With basic MyISAM on a fast server, I’m running 10GB table scans in under a minute, but moderate aggregations take a few minutes. Architecturally, MySQL is limited. One query = one thread = one core. Running two simultaneous queries is an option, but MySQL still would not do the kind of transparent, optimized caching that you need for a warehouse. Throughput is limited to disk I/O speed. InfoBright has built a column-store storage engine for MySQL but it’s targeted for the enterprise only.

What about the front end?

For the money and quality and ease of integration, it’s hard to beat Tableau. $1800 bucks isn’t cheap, but for a small business that truly needs to analyze patterns, this will do the job and it makes very pretty charts. The most recent version has integrated support for mapping based on zip code, area code, state, country and others. The maps also incorporate Census and USGS data and are pulled live from an online source. They look great! Tableau has always had a smooth, easy-to-understand layout and a crisp look that makes each chart very attractive in a presentation. It also automatically guesses what chart you want based on the quality & number of aggregates and dimensions.

The drawback is that Tableau doesn’t have its own high-speed database or ETL tool. Tableau can’t shine until a low/no-cost read-optimized database is available. Until then, it does support the most common databases and data warehouses, both commercial and open-source. Except it can’t handle generic ODBC and I don’t know why.

There’s JasperSoft = CrystalReports + OLAP + Informatica + Web Dashboards. Each component is from a different opensource project, so they don’t all use the same platform or interface, and they can’t all read the same data sources. The democratization of BI is NOT going to come from enterprise tools made cheap; it will come from simple disruptive tools that add new ideas and polish with each release. Sorry, Jasper.

What would I use to build a reporting system for a smaller business?

Well, assuming we’re doing it to make more money, not to keep up appearances, the best choice is still to pay the money for QlikView. It reads ODBC, OLE DB, text files and Excel–everything a business needs. The ETL language is easy to understand for any businessperson that has put together an Access database or enjoys Excel formulas (blech!). The GUI front-end designer is powerful & straightforward. And the in-memory database behind QlikView is so incredibly fast that I routinely analyze 10 million of rows in a split-second. It’s a one-stop shop.

Tableau is a good option but you lose the database and ETL. Maybe you don’t have a large volume of data or maybe it’s all in one view in the database–Tableau could work for you.

At a lower cost? Well, it definitely comes down to tradeoffs in coder skill, money, development time and ease of use. Whereas in QlikView anyone can write the basic code to read a couple tables, all other solutions demand heavy lifting somehwere.

If I was doing it for free?

I’d start with PHP, and possibly Ruby. Read from a database, calculate, generate Google Charts, and maybe use one of the low/no-cost Flash-based charting libraries for interactive splash. In a future post I’d like to cover ORMs and Google Chart APIs and how it can help get these projects off and running quickly.

Got any ideas? I’m always on the lookout for a faster cheaper better way to create these solutions.

50 Cool Things You Can Do with Google Charts

NYTimes Info-Graphic

Jay Jakosky — Fri, 05 Sep 2008 17:54:35 +0000

Link Here

American Life Info-Graphic For The Past Five Years — “Five Years Of Consequence”

Create a Bullet Graph In QlikView + Video

Jay Jakosky — Sat, 03 May 2008 06:02:23 +0000

What settings do you use for the gauge and bar charts? Watch the video!

Stephen Few, who spoke at the QlikView conference in April, devised the bullet graph a few years ago. A QlikView customer used bullet graphs and sparklines and was very generous to allow QlikTech to post a working demo of their application. I’m going to build the bullet graphs from that app. You can download and dissect a QVW copy of that app from the QlikView demo website.

The bullet graph in QlikView is a bar chart overlayed on a gauge chart. The demo app uses a technique of aligning the targets on all the graphs to 100% of current year budget. The formula for the black line is current year actuals divided by current year budget. The darker gauge section shows prior year actuals over current year budget.

This technique has several advantages:

Because PY and CY actuals are both divided by CY budget, they are still in harmony. You can visually see that current year sales is significantly less than last year’s sales.
Actual divided by Budget unifies many measures with wildly different scales, making chart maintenance easier without hurting accuracy.
Without this technique, you would need to write expressions for the gauge chart expression, and maximum values for bar and gauge charts. With this technique, they are 1 and 1.5.
There is additional context in answering the question, “If we were repeating last year’s performance, would we be beating our budget, and by how much?”

I hope you’ve enjoyed this tour through bullet graphs. Take a look at the demo app for sparklines in action as well.

What settings do you use for the gauge and bar charts? Watch the video!

Bullets from Few Rendered in QlikView

Jay Jakosky — Wed, 06 Feb 2008 05:39:37 +0000

Stephen Few in his blog recently highlighted bullet graphs.

I wanted to point out a QlikView demo application that uses this technique alongside Tufte’s sparklines. You can download and play with this application or examine the AJAX version.

Economist InfoGraphic of the Day

Jay Jakosky — Wed, 24 Oct 2007 18:59:12 +0000

Every day, the Economist site adds an original chart.

No related posts.

Hans Rosling Is My Hero

Jay Jakosky — Fri, 05 Oct 2007 06:16:00 +0000

I just can’t get enough of the Gapminder software, officially called Trendalyzer. The easy interface is just an enhanced scatter plot but it does the job perfectly. I think a lot of people, like me, were amazed and excited by his talk at TED 2006. He demonstrated that analytics offers real insight and improves efficiency. Now there are more “Gapcasts” to enjoy.

Interactive Information Visualization

Jay Jakosky — Mon, 02 Jul 2007 04:51:42 +0000

Enrico Bertini at Visuale asks how important is interactivity in information visualization? As a proponent of QlikView, Spotfire, Tableau and others, I think it’s extremely important. Interactivity is the future, it’s “make or break.”

I’ve been implementing speed-of-thought interactive BI tools for 6 years and I don’t want to do it any other way. When I watched my first seasoned executive lose restraint and laugh uncontrollably as he got instant answers to his hardest questions, I knew this was the only way to go. When my end-user training sessions end late because everyone is so excited about what they can do, it’s clear that people NEED interactivity.