The PC3-10600 memory is twice as fast as the 5300 memory being offered with the Xeon 7400 series processors. This is due to the introduction of the QuickPath Interconnect (QPI). The memory sticks peak at 10.6 GB/s of transfer and the QPI can support more than 25GB/s.

Also included in the Nehalem cores (i5, i7 and Xeon 7500) is a feature called Turbo Boost. Even though it reminds me of the “Turbo” button on the front of my desktop PC from the 1980’s, it is actually a feature that automatically overclocks the processor if the chip’s power and temperature are within limits. It sounds perfect for QlikView, which leaves processors sitting idle and cool, then suddenly requires a burst of the highest possible performance.

Some other features:

• 64 DIMM slots for up to 1 terabyte of memory.
• 4, 6 or 8 cores… 8 cores x 4 sockets = 32 cores
• 12, 18 and 24MB caches
• 1.86 GHz – 2.26GHz (The 2.66 GHz is being produced but not included in systems as of May 5, 2010.)
• 10 GB network connections

These are some amazing machines for running QlikView. And this ties in perfectly with recent announcements about support for large data volumes in QlikView 10, both during the load and in the server.

Below are some samples of current hardware pricing (no software, no networking, basic Dell build) as of May 5, 2010:

Systems (without memory):

• 4 processor, 8 core Xeon X7560 @ 2.26 GHz: $33,000
• 4 processor, 6 core Xeon E7530 @ 1.86 GHz: $19,500
• 2 processor, 4 core Xeon E7520 @ 1.86 GHz: $17,000

Memory:

• 128GB: $8,000
• 256GB: $14,000
• 512GB: $34,000
• 1TB: $93,000

A final note: Once again, Intel is including hyperthreading in the Nehalem cores. Hyperthreading is designed to help deal with unoptimized applications and the limitations of operating system schedulers. My understanding from the last time that hyperthreading was actively marketed is that QlikView does not benefit and can actually suffer when hyperthreading is enabled. QlikView has highly optimized code and uses it’s own threading algorithms to maintain peak performance. Hopefully someone from QlikTech can confirm in the comments that hyperthreading is not advisable.

Happy QlikViewing!

Related posts:

1. The Landscape of Parallel Computing Research
2. QlikView in the Cloud

I was building a QlikView application for financials. My client and I had discussed the idea of this application a year earlier but it was impossible then. Version 8.5 had not been released. The ability to look at dozens of simultaneous selection sets is the key to making this great idea work.

Zoom forward to 2009. Versions 8.5 and then 9.0 are released with features including Set Analysis, unlimited rows, chaining of document selections, data export from the script, and Dynamic Tables. These innovations remove the architectural limitations of QlikView that had tied my hands. A year ago I could not deliver the solution that was in my head and that my client needed. Now these limits are gone and I can build exactly what my client needs.

Build exactly what my client needs? This is the first time that this thought crossed my mind. It’s true! With the release of version 9, QlikView has entered a new phase. One that no other product can match.

QlikView is the first and ONLY tool on the market in which every business analysis that I have been asked to build can be built with confidence and an expectation of success. Dream big!

QlikView is not SPSS or JMP, and it never will be, but since I have never been asked to do anything more complex than a regression, QlikView works perfectly.

QlikView is the tool to turn to. It delivers results. Real value, right now. And you can be confident that it can achieve any business analysis you can think of. To get an idea of what QlikView can do, follow this thread on LinkedIn with over 100 unique uses for QlikView.

Related posts:

1. QlikView 9 Beta!!!
2. QlikView 8.5 Introduces “Set Analysis”
3. Seeing Dollar Signs in QlikView 8.5

I exported the blank “New” document to XML. Click here to see what it looks like in XML.

Why add XML export? One of the big-picture features that was missing from QlikView is support for version control. By using XML as the storage format you can use standard version control products and produce accurate differentials.

But now that QlikView has embraced an open file format, there are so many more things we can do…

Track Metadata in Large Deployments

Some QlikView deployments have many HUNDREDS of QVW files. Someone is probably over 1000 by now. These are unique apps; not the result of Publisher distribution loops. It’s a tedious task to manage metadata when it’s locked inside a proprietary file format.

Which apps use the [Salesman] field? Which apps have expressions that fail to multiply [Revenue] by the [Exchange Rate]? Use a tool like Rob Wunderlich’s on the XML versions of all these QVW files and we can track, compare and search hundreds of thousands of expressions, tens of thousands of layout objects, and thousands of scripts as quickly as we would explore any other data in QlikView.

Generate New Applications

Kalido is exporting basic QVWs and QVDs, That feature required custom code integration. Soon any one of us can achieve the same end result: export a complete XML specification, import the XML into QlikView, and hit reload to execute the script. If QlikTech can make this happen from the command-line then the entire process can be hidden and the end user receives a freshly baked QVW (or QVD!).

Dynamically Update Applications

You could generate a full application, but it’s more likely that you will want to make small changes to one that’s already developed.

What do you do when you cannot predict the format of a data update. What if there are new columns added each day? Or what if the headers change? One of the simple things that QlikView can’t handle on its own is the addition or removal of fields from Excel sheets. I don’t know any business intelligence tool that gracefully adapts when a data field is missing.

What if you would like to change the layout every day. Maybe a new field in the Excel sheet needs to be added to a table, or to an expression, or to a drill-down. Maybe you would like to add a chart. Every part of the application is exposed in XML, so you can do anything you want.

Go With The Flow

Speaking more generally about adapting to data… One of the mantras of business intelligence and the database world is “Will the data have this format every time?” If it won’t, no deal. Well, now we can adapt to changing data. We can also deliver dynamic applications that are driven by that data.

And more…

It’s hard to say where this goes but I think it has big potential. Are there any other BI tools that we can look to for inspiration? Do you see yourself using this feature? I’d love to read your ideas!

Related posts:

1. Using Variables In QlikView Layout
2. The Year in QlikView

You can access the beta as a registered customer or partner here: http://www.qlikcommunity.com/qv9beta/

My favorite new features so far:

• iPhone client. Zoom in. Shake to clear. Lots of fun.
• Java client for everyone else, including Blackberry.
• Removal of the 2 billion record limit. Now your record set is limited by RAM and CPU power.
• Dynamic data update. Update data in the QlikView document without a reload through APIs and standard SQL statements. When used on the server, it will PUSH the data out to the clients.
• Export QlikView documents as XML. QlikView documents (no data) can be stored in version control systems. Differentials can be generated. This also means that QlikView documents can be programmatically generated!
• Document chaining. Open one document from another and apply selections. Great for working with segmented QlikView documents when data volumes get very large. Can also be combined with the programmatic QlikView generation in interesting ways!
• Actions and Trigger Actions are replacing macros. Oh please, oh please be the end of macros!!
• Set Analysis can now reference other fields, as in “Select all possible values in Customers based on a Sales last year”.
• Sparklines. Mr. Tufte, your “tiny” contribution is everywhere. Cool.
• Trellis charts. I’m quite proud of this since it was originally my suggestion!! Woo hoo!
• Dynamic background charts. Swap out map backgrounds as selections are made.

Related posts:

1. The Year in QlikView
2. QlikView 8.5 Introduces “Set Analysis”
3. QlikView 9: Export Document Layout to XML

QlikView’s Rapid Time-to-Implementation Improves BI Value

Related posts:

1. The Year in QlikView
2. QlikView 8.5 Introduces “Set Analysis”
3. Using QlikView Tickets

The problem is virtualization. Virtualization is valuable to customers and service providers, but it’s also a thief! It adds overhead for the processor, cache and memory–everything that impacts QlikView performance!

The cloud, as in real life, is ever changing. You have no idea how many people are sharing your hardware and what their load will be from second to second. I would bet that nearly all deployed QlikView servers spend most of their time idle and the rest of their time at peak processing power. In the cloud, the goal is to spend as little time as possible idle for which we sacrifice peak processing power. QlikView depends on peak processing power and that type of application will suffer the most in the cloud.

But exactly how much will it suffer? Success in the cloud will need to be measured by the end-user experience. The cost of being in the cloud is vigilant monitoring and smart responses. What’s the right way to monitor the end-user experience in a company that uses OCX vs. AJAX, or is spread out geographically? Will bringing up more servers in the cloud improve response time? Should every QlikView server deliver the same set of apps, or should each app be served by a dynamic set of servers? Similarly, do some apps need sub-second response time while others can wait?

One thing stays the same. If you deploy large QlikView data sets you’re already sensitive to response times and what to consider when designing an app. In the cloud, smaller apps will need to think about costly chart expressions, messy data models and design choices that work fine on dedicated servers.

Related posts:

1. Vertica for the Cloud
2. QlikView Scalability
3. Using QlikView Tickets

Salesperson: “What shipped yesterday? Ok, what’s the breakdown? Woah, what happened in that department? That markdown is too steep, who wrote that order? Which customer? What’s that rep’s extension?”

With one-second results, that analysis would have happened in the time it took you to read it. This is a competition against human nature. One-second results makes the difference between wishing you had the answer and getting it, multiplied over and over throughout the day.

The impact on a business is not from faster queries alone. Behavior changes when decision makers trust that the data is immediately at hand. The relationship to data changes when you can find the answer while you think about it and not lose your train of thought.

Because the query engine can respond to any query in one second, we can make every path of exploration available at the beginning. One application can take the place of many reports. Users can begin to query immediately and along any drill path. The benefit of one-second results is diminished if users have to first identify the report that has the data and filtering options they need.

Can OLAP deliver this? No. We must combine speed of execution with rapid application development, full transaction details, and eliminate predefined drill paths. OLAP/MOLAP/ROLAP/SCHMOLAP can’t take us into this new era. In-memory associative and column-store databases can.

With one-second results, you don’t build a query and then start the execution. Instead, the results update as soon as you pick the first filtering option, whether it’s the day, order number or country of origin. You get immediate feedback before you make your next selection. Also, the filter options can change based on the results. Maybe you remove options that are incompatible with the selections made so far. By shrinking the feedback loop with one-second results, the filtering options can show intelligent behavior to help guide users or add context to the results. This level of dynamism lets users roll back and forth through their ideas. They can cross-reference without losing a train of thought, or discover and follow tangents that are more important.

It’s not just one decision maker getting an answer quickly. Interactions and processes benefit. Workers get feedback in near-real-time. We can do tricks like running the same query once per second. Ridiculous? This isn’t paradise, I live in the land of low budgets and “getting it done”. Vendor and customer data is available right when they’re on the phone. Less “I’ll get back to you” and more “I have that info right in front of me.” I’ve also noticed that it’s harder to bullshit when anyone in the meeting can easily explore the data on their laptop and get the real answer.

In companies where I can deliver one-second results, I spend a lot of time reconditioning people to ask for anything they desire, because now I can put any information at their fingertips, no matter how many tables, how much detail and with little knowledge of how they want to look at the data.

For nearly all companies, the entire transactional database can be copied as-is into a one-second query engine. Add a BI tool on top, rename some fields and identify the table relationships. Time is spent developing the frontend to deliver the best reports and analysis. One person can build the entire solution. Since the transactional model is already validated, there is no data modeling, no formal architecture and little documentation. This might be frightening to enterprises but the benefits are huge for strapped IT budgets.

A one-second query engine needs an interactive frontend to take advantage of it. We also need simpler ETL tools. With the engine in place first, developers will connect the dots and the tools will be built to take advantage of the new abilities.

None of this is theoretical. I’ve been doing this for the past 7 years with an in-memory associative database, ETL tool and interactive frontend called QlikView. When information flows at the speed of thought, it changes decision-maker behavior and the business process. When we can prototype and deploy one-second query engines quickly, then ideas can be built and tested quickly. Most ideas won’t be new or unexpected, but they were impossible or impractical without one-second results.

Related posts:

1. Peter Batty Discusses One-Second Results And Geospatial Analysis
2. Low-Cost Data Analysis & Visualization: It’s Getting Better All The Time
3. What makes QlikView so good?

My test data set is 37 million rows of point-of-sale transactions. Total data size as CSV is 7GB. My test system stinks. I need to make that clear so that my numbers are not seen as representative of what’s possible with InfoBright. After seeing the product in action, I’m sure that server hardware will do much better.

How fast to bulk load?

InfoBright loads are multi-threaded, but my test server is a single-processor desktop and the loads are still fast! With my single processor, about 1.8 million rows/minute (336 MB/min) are being inserted and the load rate slowed down about 10% over 37 million rows. Disk access was minimal as records were inserted. Overall, my little desktop moved an average of 30,000 rows/sec or 5.6 megabytes/sec. That’s 20GB/hour! My processor was fully loaded every second. With faster cores and multi-threading, the load should be much faster. When I get the chance to load Linux on a bigger box I’ll be eager to see how it performs.

How big on disk?

I have 7GB of data. Using MySQL’s default MyISAM storage engine with an 8-bit ASCII representation requires… 7GB. No surprise there. InfoBright took 591.2MB, as reported from my MySQL management console. That’s a 92% reduction in size or a 12:1 compression ratio.

The status data coming from the InfoBright engine includes the storage size of each column and total size of the table. If I could remove the lowest-level detail, InfoBright reports exactly how much space that would save. Helpful.

How much memory?

I don’t have much guidance because I don’t have enough data to stress the cache. My largest data set can fit comfortably inside the compressed cache. That means every company I’ve ever dealt with would be able to avoid disk reads and improve performance. Unfortunately, this does not put InfoBright’s performance on par with other in-memory databases. More on this later.

Here are some guidelines from InfoBright on the memory (in megabytes) that you should allocate given a certain amount of system memory. These figures have no relationship to the size of your data set. I also don’t know if 32 GB represents an upper limit for the InfoBright software. I suspect the point to this table is that the loader heap does not need to increase and that the compressed heap should increase the fastest but will not exceed the main heap.

ServerMainHeapSize – Size of the main memory heap in the server process, in MB
ServerCompressedHeapSize – Size of the compressed memory heap in the server, in MB.
LoaderMainHeapSize – Size of the memory heap in the loader process, in MB.

Performance?

Is it fast? Slow? My hardware is too restrictive to see what InfoBright can do. All signs are promising. What I can say is that the cache grew over time until MySQL was barely touching the disk. My processor is completely peaked, with 99.8% allocated to the MySQL process. According to this article published by MySQL yesterday, InfoBright queries are (for now) restricted to one CPU core. Performance is dependent on the size of my cache and the speed of each core, two things I have direct control over.

Even with my little desktop testbed, this much is clear: the QlikView in-memory database is much faster. On this dataset I’d see results in a split-second instead of 30, 60 or 120 seconds. You might think that comparing these two products isn’t fair, but if your goal is to deliver analysis in SMEs or enterprise departments, these two will definitely compete and complement one another.

Summary?

One of the advantages of column-stores for data warehousing is that simply replicating the original transactional schema can yield adequate performance. Also, there is no performance hit for bringing in the lowest level of granularity. With column-stores, you may not need to build snowflake schemas or do much transformation. Column-stores are therefore less effort to get started in smaller companies with resource-starved IT departments. This means a faster failure rate which is what interests me most: implement quickly, measure early impact and choose investment (InfoBright Enterprise), deferral or elimination.

There is one other free column-store database of significance, MonetDB. It’s an academic project and as such it lacks the toolset and polish that InfoBright inherited from MySQL. I was up and running faster with InfoBright than I was with MonetDB because the installers and administration utilities for InfoBright are already familiar. My Windows tools for MySQL connected right in without a problem. My front-ends with simplified MySQL connectors were oblivious to the InfoBright backend, which is absolutely how it should be.

InfoBright is not without its issues. Documentation is thin or non-existant. I spent hours and hours until I determined (and confirmed on the forums) that the InfoBright loader does not support all of the MySQL syntax for bulk loads. This would not have been such a problem if the error message had provided some warning about my syntax that was perfectly legal in standard MySQL.

All in all, I’m thrilled to have a no-cost column-store database available for prototyping, quick and dirty applications, and bulk data storage.

Related posts:

1. Low-Cost Data Analysis & Visualization: It’s Getting Better All The Time
2. Infobright 3.0.2 Released
3. QlikView in the Cloud

Of course, I could just buy QlikView, but what can be done for less $money? Unfortunately data warehouses and BI front-ends are not sexy problems in the opensource community. Graphs and charts get a little more attention, but you’ll need to write your own code to glue them to your application.

In summary, what can I say about our options?

First, write your own ETL. Why do opensource ETL tools like Talend and Kettle work so hard to rebuild Informatica? It reminds me of Linux in the 1990s when the community wanted to beat Windows and kept working to look like Windows and wondering when victory would arrive. Informatica, like OLAP and mainframes, is from an era when memory was scarce; languages were low-level, slow to compile & run, abstracted little and were not at all portable. On top of that, ODBC drivers were tightly controlled and costly.

But now we can pick from many great scripting languages. Today’s languages abstract the hard parts, are easy to read, can be edited while executing and talk to any system, database, web service or application. I think the next direction for ETL will be a simple (but extensible) transformation language using an ORM wrapper… Rails on ETL. Until that arrives, you can achieve everything you need with PHP, Perl, Ruby and others.

Best option for low-cost data warehouse?

Check out the totally free MonetDB. Unless Vertica or InfoBright reconsiders releasing a low/no cost option, MonetDB will likely mature to become a first-choice column-store database. It’s an academic project that has earned a sizeable development community and user base. The product is functional today for tens of millions of rows (maybe more). So far I have personally worked with a few million rows in MonetDB and I’d like to use it again. With a little focus on usability and packaging, it could be a contender.

Greenplum, freely available for development, won’t help. The architecture is designed around Massively Parallel Processing. As a single, standalone installation, it’s basically just PostgreSQL. You won’t see extra performance without a farm of servers.

To my surprise, MySQL itself is not too bad. The MyISAM tables are speedy and Alex Tomic wrote a post about using multiple queries against the Archive storage engine and how to steal an index with that engine. With basic MyISAM on a fast server, I’m running 10GB table scans in under a minute, but moderate aggregations take a few minutes. Architecturally, MySQL is limited. One query = one thread = one core. Running two simultaneous queries is an option, but MySQL still would not do the kind of transparent, optimized caching that you need for a warehouse. Throughput is limited to disk I/O speed. InfoBright has built a column-store storage engine for MySQL but it’s targeted for the enterprise only.

What about the front end?

For the money and quality and ease of integration, it’s hard to beat Tableau. $1800 bucks isn’t cheap, but for a small business that truly needs to analyze patterns, this will do the job and it makes very pretty charts. The most recent version has integrated support for mapping based on zip code, area code, state, country and others. The maps also incorporate Census and USGS data and are pulled live from an online source. They look great! Tableau has always had a smooth, easy-to-understand layout and a crisp look that makes each chart very attractive in a presentation. It also automatically guesses what chart you want based on the quality & number of aggregates and dimensions.

The drawback is that Tableau doesn’t have its own high-speed database or ETL tool. Tableau can’t shine until a low/no-cost read-optimized database is available. Until then, it does support the most common databases and data warehouses, both commercial and open-source. Except it can’t handle generic ODBC and I don’t know why.

There’s JasperSoft = CrystalReports + OLAP + Informatica + Web Dashboards. Each component is from a different opensource project, so they don’t all use the same platform or interface, and they can’t all read the same data sources. The democratization of BI is NOT going to come from enterprise tools made cheap; it will come from simple disruptive tools that add new ideas and polish with each release. Sorry, Jasper.

What would I use to build a reporting system for a smaller business?

Well, assuming we’re doing it to make more money, not to keep up appearances, the best choice is still to pay the money for QlikView. It reads ODBC, OLE DB, text files and Excel–everything a business needs. The ETL language is easy to understand for any businessperson that has put together an Access database or enjoys Excel formulas (blech!). The GUI front-end designer is powerful & straightforward. And the in-memory database behind QlikView is so incredibly fast that I routinely analyze 10 million of rows in a split-second. It’s a one-stop shop.

Tableau is a good option but you lose the database and ETL. Maybe you don’t have a large volume of data or maybe it’s all in one view in the database–Tableau could work for you.

At a lower cost? Well, it definitely comes down to tradeoffs in coder skill, money, development time and ease of use. Whereas in QlikView anyone can write the basic code to read a couple tables, all other solutions demand heavy lifting somehwere.

If I was doing it for free?

I’d start with PHP, and possibly Ruby. Read from a database, calculate, generate Google Charts, and maybe use one of the low/no-cost Flash-based charting libraries for interactive splash. In a future post I’d like to cover ORMs and Google Chart APIs and how it can help get these projects off and running quickly.

Got any ideas? I’m always on the lookout for a faster cheaper better way to create these solutions.

50 Cool Things You Can Do with Google Charts

Related posts:

1. Interactive Information Visualization
2. Response to the Tableau 3.0 Webinar
3. What’s Vertica?

ODBC is a slow protocol, running orders of magnitude slower than the database or a typical Ethernet connection. Very pricey ETL tools for data warehousing get around this by extracting through multiple connections to the database, and there’s no reason that a QlikView infrastructure can’t take advantage of it.

For example, run two copies of QlikView at the same time and extract approximately half of the data set with each. First, make a copy of the QV.exe file and give it a unique name. You can open QV.exe and your unique copy at the same time. You can run three or more copies of QlikView with this method.

Next, decide how to divide your data set; it could be based on date, country, state, or half the alphabet, for example. What you want is to divide the data set into roughly equal segments, one for each copy of QlikView.

How does each copy of QlikView know which segment to load? One way to do this dynamically is to use the command-line to set a variable in the script. Reference this variable in the SQL SELECT statement in the script: WHERE YearField=$(vYearVariable). See the reference manual for command-line options.

Your mileage will vary. Some databases don’t do much better with simultaneous ODBC reads. Oracle does quite well.

Related posts:

1. Faster Reloads With “Core Control”
2. Seeing Dollar Signs in QlikView 8.5
3. QlikView 8.5 Introduces “Set Analysis”