... and points beyond » QlikView

Great Features In QlikView 11

Jay Jakosky — Thu, 20 Oct 2011 23:22:47 +0000

QlikView 11 looks great. Three new features in particular are going to make great impressions: the improved Web interface (AJAX), Session Sharing, and Notes.

No related posts.

Writeback in QlikView

Jay Jakosky — Sat, 02 Jul 2011 22:45:57 +0000

The ability to push data from QlikView back out to a database is beneficial for what-if analysis, financial reporting, CRM dashboards and more. Unfortunately our ability to achieve this, even using custom code, is limited. On the frontend, there are only a few objects that can support user input and very little control over how these objects display. On the backend, connecting QlikView back to a database is very difficult to do well.

Part of the problem is that QlikView server is not always in control of this communication. If you’re using Plugin, it’s done by the client. If you’re using AJAX or Mobile, it’s done by the server. If, however, you code your solution as an Extension, things shift back to the client side again.

The common way to implement writeback is through VBScript macros. Examples of code to achieve this are plentiful on QlikCommunity. Although a simple version can be mocked up in a few lines of code, deploying this solution to many users in a modestly secure environment has serious disadvantages.

Each client machine needs to communicate to the database. Therefore, database drivers need to be installed on each client. Credentials need to be included in the code of each QlikView document, leaving them exposed to users. Special ports need to be open in firewalls for driver communication. These are poor security practices and should be reason enough for any enterprise to abandon this approach.
Client-side code is difficult to monitor. Error handling is poor. A separate system would need to be in place to capture errors for analysis and resolution.
Managing conflicts in a distributed environment requires careful design and development.
Communication between the VBS execution environment and the database can be slowed for any number of reasons. This leaves the client in an uncertain state, without feedback on progress or problems. Meanwhile, the application state in QlikView can continue to change. This easily can cause inconsistencies.
Database driver communication uses proprietary protocols that are difficult to monitor for debugging and by network security software.

But there is a much better way to implement writeback to a database from QlikView: build a lightweight web service. What this means is to have QlikView send a structured request to a web server that can interpret the request, make the appropriate database changes, and send a useful response back to QlikView. Overall, this approach is far more flexible, reliable, compatible, configurable and maintainable.

The response to a web service command (HTTP POST) can itself be an extensive report on the success or failure of any updates. This data can be made visible to the user as a clear confirmation that changes were successful.
Server-side code is more reliable. It’s far easier to manage many users updating data at the same time. It’s easier to record and react to errors. Implementing your web service in PHP gives you a community with examples of good design.
Server-side code can handle any level of complexity such as triggering other systems. Client crashes need not leave complex processes in unresolved states.
This approach only requires a web port to be open in the firewall and therefore is more likely to work regardless of where the user is located.
With this approach, it is easier to handle database rollbacks, atomic transactions and other features that support the completion of a transaction.
Changes do not need to be sent to a database one value at a time. Instead, changes can be aggregated into a single update on the QlikView side. Aggregating changes is done faster than database communication. There is less chance of stalling the user session or allowing QlikView’s state to change in the middle of an update process.
Multiple tables can be updated. For example, not only can a value be updated, but a separate audit log can be updated with who made the change, when, and to what value.
Communication using XML over HTTP is readily captured by network security software.
Web services can leverage existing network infrastructure. For example, IIS & Active Directory will authenticate the user making the web request. The web service code can be passed this information reliably.
The database is read and written by a single set of credentials, written once in the web services code, and running on a secured server, without any access from other machines on the network. This is far more secure and a much easier sell to the IT department.

I’ve had plenty of success with this approach, combining IIS, ActiveDirectory, PHP and QlikView VBScript macro code. I don’t think we need writeback as a QlikView feature. I would, however, like to see a few changes to QlikView to better support features like this.

Support the editing of Input Fields in more objects, such as when used as dimensions in a Pivot Table, or in a bar chart.
Support multi-line text in Input Fields.
Add functions to VBScript and Extensions to identify the rows of a table with Input Fields that have been changed since reload. For Extensions, something like a “next” iterator that moves to the next changed value.
Make it possible to share Input Fields across users–without using collaboration objects as a kludge.
Add Extension/AJAX functions for managing the data behind Extension objects with millions of rows.
Support the updating of Input Fields from AJAX.

Happy Qliking!

The Year in QlikView

QlikView Google Maps Javascript v3 Integration

Jay Jakosky — Fri, 24 Jun 2011 19:32:11 +0000

Here is a sample of integrating a Google Maps JavaScript v3 API map into QlikView as an extension object.

QlikView Integrate Google Maps JavaScript v3 API

Copying and Pasting Colors In QlikView

Jay Jakosky — Fri, 20 May 2011 21:46:05 +0000

It is possible to copy and paste color settings among all layout objects! Even two-color gradients can be easily copied and pasted. If the destination object doesn’t support gradients, the first color is kept and the second color is ignored.

Simply right-click on the color box to copy or paste.

My thanks to Joe Feyas for discovering this!

Can a humble Chart object get some love?

Jay Jakosky — Fri, 01 Apr 2011 15:09:03 +0000

If there was just one question I could ask at this year’s Qonnections 2011, it would be this…

When are we going to see improvements to the most basic QlikView task: displaying data?

Look at the following examples from competitors…

Above is a Spotfire chart that cleanly displays a 2-level hierarchy of dimension values on the x-axis. Increase to 3 levels and the labels stay organized and readable.

Below is a chart from Tableau.

The axis labels are only shown at the left and the bottom of the entire trellis. QlikView shows axis labels on each square, adding unnecessary clutter that is not easy to remove. Two dimension values are coded in the size of the dots and their color. Tableau also uses color gradients easily and effectively.

Tableau and Spotfire put a lot of energy into making displays clean and readable. Tableau makes excellent guesses at how to display your data.

QlikView’s charts have felt clunky for years. The Chart building dialog is huge, confusing and too often doesn’t work as expected. Charts don’t adapt well to being small. Axis labels cram into each other, don’t split lines, and don’t respect chart settings. Legends use excessive real estate, have limited positioning with no intelligence and don’t split text. Expression cycles are confusing for end-users. Fonts and colors are buried 3 levels deep. “Themes” exclude certain chart elements, requiring the developer to dive deep into menus to make targeted changes. Scatter plots quickly become a messy jumble of points and labels. Removing scatter plot data point labels makes identifying a data point a painful task of color matching.

There doesn’t seem to be any point in discussing geospatial data, for which QlikView has no native abilities. QlikTech has been frustratingly quiet on this. Want to include Google Maps? You’re welcome to search for code in the community, or pay more for third-party tools. Meanwhile, the competitors’ native support is easy and attractive.

QlikView is still the best tool out there for “getting things done”. Graphical display is one of a few areas where QlikView is lagging. But QlikView is too far behind at this point. Charts have not been overhauled since at most version 7. It’s time for a major leap forward.

Trigger QlikView Publisher EDX Task From Windows Powershell

Jay Jakosky — Fri, 01 Apr 2011 04:04:09 +0000

Here’s a script to trigger an EDX task from Powershell. As it is, you will need to change QVSERVER to match your server name. The script can then be run from the command line by passing the task name and EDX password as parameters.

Download the QlikView EDX Trigger in Powershell. Do not copy and paste the code below.

param($taskName,$taskEDXPassword)

function QVPOST([string]$updateurl, [string]$text)
{
     $result = $null
     [System.Net.HttpWebRequest] $request = [System.Net.HttpWebRequest] [System.Net.WebRequest]::Create($updateurl)
     $request.UseDefaultCredentials = $true
     $request.Method = "POST"
     $request.ContentType = "application/x-www-form-urlencoded"
     $request.ContentLength = $text.Length

     [System.IO.StreamWriter] $stOut = new-object System.IO.StreamWriter($request.GetRequestStream(), [System.Text.Encoding]::ASCII)
     $stOut.Write($text)
     $stOut.Close()

     [System.Net.HttpWebResponse] $response = [System.Net.HttpWebResponse] $request.GetResponse()
     if ($response.StatusCode -ne 200)
     {
           $result = "Error : " + $response.StatusCode + " : " + $response.StatusDescription
     }
     else
     {
           $sr = New-Object System.IO.StreamReader($response.GetResponseStream())
           $result = $sr.ReadToEnd()
     }

     return $result
}

1$response = 1(QVPOST "http://QVSERVER:4720/qtxs.asmx" "")
$requestKey = $response.GetTimeLimitedRequestKey.GetTimeLimitedRequestKeyResult

$taskEDXRequest=@"

$($taskName)
$($taskEDXPassword)



"@

$response = QVPOST "http://QVSERVER:4720/qtxs.asmx" $taskEDXRequest
echo $response.RequestEDX.RequestEDXResult

A Brief Look at QlikView Storage

Jay Jakosky — Tue, 05 Oct 2010 03:26:31 +0000

You can learn a lot about how QlikView stores data in memory by looking at how it stores data on disk in QVDs.

QVDs are QlikView’s proprietary data storage format. QVDs are in a format optimized for reading into memory. A QVD stores one QlikView table. Each column is stored separately. Only the unique values are stored for each column; this mimics QlikView’s storage of unique values in memory. With all the column values stored the only thing missing is a data structure–one record for each row of the table–that stores a series of indexes to the unique value stored in each field of the row. In fact these indexes are stored in a highly compressed format that mimics the storage of indexes in memory.

Note: By using QVDs, the storage of the overall table is much smaller, but the unique column values are not compressed using something like ZIP or Gzip compression. Why? Because that would make the loading of QVD files slower due to the overhead of decompression! Try compressing a QVD sometime. They get a lot smaller!

Getting back to the point of this post… In the header of a QVD file is some valuable information that you can use to measure the size of data in memory. Open up your QVD using a text editor. If your QVD is very large, you will need an editor that can handle large files. Look at the tags that are in your table. The structure is similar to the following:

Date

Within each FieldName tag, you will see and . Length is how much memory is needed to store the columns’ unique values in memory. Length / NoOfSymbols = bytes per symbol, which you can use to estimate your in-memory storage needs as data volumes grow.

In the real world, your in-memory storage may be better than a QVD. QlikView has optimizations that can dramatically improve storage. They are not used in QVDs because they either slow down the loading of data from QVD or because it is not possible to perform the optimization until the QlikView Script has finished execution. For example, QlikView will store a column of entirely consecutive values (11, 12, 13…100) as offsets from the base value (11) rather than using 8 bytes to store each unique value. This optimization can’t be done until the script is finished and QlikView can evaluate the column in its final form.

I hope this helps you get more information on how QlikView is handling your data. Some of the topics related to this post would be: (a) the storage of numbers as both text and numeric representations, how to identify this and how to avoid it; and (b) more definitive calculations for estimating your storage needs in-memory with QlikView Server.

UPDATE: QlikView 10 treats numbers differently in memory versus in QVD. QV 10 does not store numbers in memory with fewer than 9 bytes (8 bytes + 1 byte of overhead). However, QV 10 does store numbers in QVDs using 5 bytes (4 bytes + 1 byte of overhead) when the values meet certain criteria.

InfoBright Open-Source Column-Store DBMS

Where Have The QlikView Bloggers Gone?

Jay Jakosky — Fri, 24 Sep 2010 22:54:34 +0000

Are we waiting for QlikView 10? It’s so quiet out here in the blogosphere.

No related posts.

QlikView Scalability

Jay Jakosky — Wed, 05 May 2010 20:03:16 +0000

QlikTech recently announced their Scalability Center for customers to evaluate their QlikView apps under load and using the best servers available. At the time of the announcement, just one day after Intel’s own, they say they will have systems based on the Xeon 7500 series processors. The performance improvements over the 7400 series alone are impressive. Rather than look at what the processor and chipset can do in the Intel literature, let’s look at what they are doing in offerings from Dell and IBM.

The PC3-10600 memory is twice as fast as the 5300 memory being offered with the Xeon 7400 series processors. This is due to the introduction of the QuickPath Interconnect (QPI). The memory sticks peak at 10.6 GB/s of transfer and the QPI can support more than 25GB/s.

Also included in the Nehalem cores (i5, i7 and Xeon 7500) is a feature called Turbo Boost. Even though it reminds me of the “Turbo” button on the front of my desktop PC from the 1980′s, it is actually a feature that automatically overclocks the processor if the chip’s power and temperature are within limits. It sounds perfect for QlikView, which leaves processors sitting idle and cool, then suddenly requires a burst of the highest possible performance.

Some other features:

64 DIMM slots for up to 1 terabyte of memory.
4, 6 or 8 cores… 8 cores x 4 sockets = 32 cores
12, 18 and 24MB caches
1.86 GHz – 2.26GHz (The 2.66 GHz is being produced but not included in systems as of May 5, 2010.)
10 GB network connections

These are some amazing machines for running QlikView. And this ties in perfectly with recent announcements about support for large data volumes in QlikView 10, both during the load and in the server.

Below are some samples of current hardware pricing (no software, no networking, basic Dell build) as of May 5, 2010:

Systems (without memory):

4 processor, 8 core Xeon X7560 @ 2.26 GHz: $33,000
4 processor, 6 core Xeon E7530 @ 1.86 GHz: $19,500
2 processor, 4 core Xeon E7520 @ 1.86 GHz: $17,000

Memory:

128GB: $8,000
256GB: $14,000
512GB: $34,000
1TB: $93,000

A final note: Once again, Intel is including hyperthreading in the Nehalem cores. Hyperthreading is designed to help deal with unoptimized applications and the limitations of operating system schedulers. My understanding from the last time that hyperthreading was actively marketed is that QlikView does not benefit and can actually suffer when hyperthreading is enabled. QlikView has highly optimized code and uses it’s own threading algorithms to maintain peak performance. Hopefully someone from QlikTech can confirm in the comments that hyperthreading is not advisable.

Happy QlikViewing!

QlikView in the Cloud

The Year in QlikView

Jay Jakosky — Fri, 08 Jan 2010 23:06:49 +0000

A subtle and powerful shift happened last year.

I was building a QlikView application for financials. My client and I had discussed the idea of this application a year earlier but it was impossible then. Version 8.5 had not been released. The ability to look at dozens of simultaneous selection sets is the key to making this great idea work.

Zoom forward to 2009. Versions 8.5 and then 9.0 are released with features including Set Analysis, unlimited rows, chaining of document selections, data export from the script, and Dynamic Tables. These innovations remove the architectural limitations of QlikView that had tied my hands. A year ago I could not deliver the solution that was in my head and that my client needed. Now these limits are gone and I can build exactly what my client needs.

Build exactly what my client needs? This is the first time that this thought crossed my mind. It’s true! With the release of version 9, QlikView has entered a new phase. One that no other product can match.

QlikView is the first and ONLY tool on the market in which every business analysis that I have been asked to build can be built with confidence and an expectation of success. Dream big!

QlikView is not SPSS or JMP, and it never will be, but since I have never been asked to do anything more complex than a regression, QlikView works perfectly.

QlikView is the tool to turn to. It delivers results. Real value, right now. And you can be confident that it can achieve any business analysis you can think of. To get an idea of what QlikView can do, follow this thread on LinkedIn with over 100 unique uses for QlikView.