ODBC is a slow protocol, running orders of magnitude slower than the database or a typical Ethernet connection. Very pricey ETL tools for data warehousing get around this by extracting through multiple connections to the database, and there’s no reason that a QlikView infrastructure can’t take advantage of it.

For example, run two copies of QlikView at the same time and extract approximately half of the data set with each. First, make a copy of the QV.exe file and give it a unique name. You can open QV.exe and your unique copy at the same time. You can run three or more copies of QlikView with this method.

Next, decide how to divide your data set; it could be based on date, country, state, or half the alphabet, for example. What you want is to divide the data set into roughly equal segments, one for each copy of QlikView.

How does each copy of QlikView know which segment to load? One way to do this dynamically is to use the command-line to set a variable in the script. Reference this variable in the SQL SELECT statement in the script: WHERE YearField=$(vYearVariable). See the reference manual for command-line options.

Your mileage will vary. Some databases don’t do much better with simultaneous ODBC reads. Oracle does quite well.

Related posts:

1. Faster Reloads With “Core Control”
2. Seeing Dollar Signs in QlikView 8.5
3. QlikView 8.5 Introduces “Set Analysis”

Load times on large datasets can be dramatically improved by splitting the reload process into pieces and assigning pieces to separate cores. A good place to start is monitoring how your Windows installation allocates processes to cores. I’ve only seen one tool that shows processes on each core. If you know of any others, please post a comment.

• CPU-Control

Now that you have evaluated the automatic allocation, how can you specify the core directly? One tool that ships with Windows is the Task Manager. Unfortunately, it can only alter a running process and it does not save your configuration for the next run. NOTE: Vista allows command-line affinity setting and Server 2008 might as well.

There are several choices of utilities to control core affinity. Some are command-line such as BeyondLogic’s, others have a GUI and store your choice for the future such as Tom’s Hardware Guide’s. The downside with THG’s tool is that you will need to make copies of QV.exe with different names so each gets a unique process name. The upside is that the CPU affinity setting is automatic.

• BeyondLogic Process Utility
• Tom’s Hardware Guide Affinity Tool
• BindCPU Utility
• ProcAff
• SysInternals’ PsExex (written for remote execution)
• ImageCFG.exe from an older Windows Toolkit

Obviously, trying to do this with Publisher would be a challenge. The upcoming Publisher 8.5 will support command-line execution which may work with these utilities, however you will lose visibility in the Publisher console. It’s all tradeoffs, but at least CPU affinity is another tool in your belt.

CPU Control

  Tom’s Hardware Guide Task Assignment Manager

Related posts:

1. Improving The Load Process With Multiple ODBC Connections
2. QlikView Scalability
3. QlikView 9: Export Document Layout to XML