IBM Blueview

Cognos Analytics and all things IBM

The Blog

Cognos 11.1.4 Performance Tips

by Leave a Comment

Are you attending Data and AI Forum in Miami next week? Want to talk some Cognos with me? Snag a half hour of my time to chat free from sales guys.

Thank you to those who attended Tuesday’s 11.1.4 livestream! Technical issues aside I had a great time doing it and will be hosting more in the future. There were some comments about performance in recent releases that I promised to address but didn’t – so here are my Cognos 11.1.4 performance tips!

Tip one: Dispatcher > Gateway

Cognos 11 brought with it the option to dispense with the gateway entirely and sending your traffic directly to the dispatcher. This is, among other things, a performance hack that can pay big dividends, particularly if your Cognos environment takes an extremely long time to load or if the UI feels generally sluggish.

The downside here is that for most enterprise deployments relying on a single dispatcher is just not an option for a ton of reasons – including little things like SSO, load balancing and disaster recovery. But if you are able to try this you might find it helps quite a bit.

Tip Two: Reality > Minimum Requirements

The minimum requirements for Cognos 11.1.4 are currently listed at two CPUs. This is a straight up no-go for using Cognos in almost all circumstances regardless of what version you’re on so long as that version starts with the number 11.

A tale of two configs

I typically deploy a single-instance Cognos machine on AWS with the following specs and have no performance issues for demos or small deployments:

  • ec2 t3.xlarge
  • 4 vCPUs @ 2.5Ghz
  • 16GB RAM

With 11.1.4 this configuration suffers from a laggy UI and poor query performance, especially in Dashboards. I observed CPU utilization hitting 100% frequently. As a remedy I migrated my AMI to:

  • ec2 c4.4xlarge
  • 16 vCPUs @ 3.4Ghz
  • 32GB RAM

This adjustment resulted in extremely snappy performance throughout the UI and like-for-like dashboard query execution improved from ~25 seconds to ~2 seconds.

Why did this jump so much?

Cognos 11.1.4 is doing A LOT more than 11.0.x or 10.x ever dreamed of with a host of new interactive services running in the background – services that do things like natural language query, automatic visualization or dashboard creation, on-the-fly predictive forecasting, integrated data prep and modeling…

Tip Three: SSD > HDD

One often unexplored area of performance improvement for Cognos 11.1.4 (or anyone using Transformer for that matter) is increasing the speed of your HDD or migrating to an SSD. All of the new features rely on either the Cognos 10 DQM engine or the Cognos 11.1 ‘Flint’ spark-sql engine to process queries. In either case data is written to and read from disk so IO becomes larger factor in determining performance than ever before, especially for any interactive task.

Tip Four: Your Browser Matters Now

Go ahead and conduct an experiment for me – pop into Dashboards on your local machine, build a bunch of visualizations and observer your resource use. You may notice that Cognos running in Chrome is capable of maxing the CPU on your laptop. It is capable of maxing the CPU on my monster gaming desktop as well if I push it hard enough.

So the final place I’d look to juice performance is actually at the hardware your users are toting around with them. It may seem a little unfair but just imagine life as a poor Power BI or Tableau administrator – outdated and slow laptops define the entirety of their user experience.

Cognos 11.1.4 Performance Summary

All of this is to say that I have had success increasing performance by significantly increasing the specs of the application server. This isn’t necessarily shocking but it’s comforting in a way because it means the problem isn’t bad code. It also means that if you’ve been skating by on the same server config you put into place in 2015 it’s probably time for an upgrade.

Join me at Data and AI Forum 2019

by Leave a Comment

I’ll be attending Data and AI Forum again this year, as I do every year. Data and AI Forum is THE conference to attend for Cognos, Planning Analytics and other IBM analytics technologies. THINK is the big IBM conference but for us, this is the valuable one.

If you’re interested in saying hello you can find me at the PMsquare booth or attend one of my sessions:

If you’re struggling to modernize your Cognos environment, the second session is a treat.

Why Gartner is Worthless

by 8 Comments

Gartner is probably your organization’s go-to resource for assessing software which is a shame because Gartner is worthless. Their process is misunderstood by the industry and their magic quadrants are utterly misleading. It’s all the worse for their ubiquity as a key reference for decision makers. Why do I feel this way? Let’s examine the recent Magic Quadrant for financial planning and analysis as a case study.

When a picture is worth no words

Let’s be honest – almost nobody reads these Gartner reports. They look at the picture and they pick the top 3-5 for an RFP figuring they have their bases covered. So looking above, I’m going to bring in Oracle, Workday and Anaplan for my new planning solution and I’m going to be biased towards Oracle as being the ‘market leader’.

What makes Oracle a market leader?

Let’s look past the MQ image though, scroll down through the report and read what they actually have to say about Oracle – the market leader! I’ll summarize below.

Strengths

  • Lots of customers
  • High brand recognition
  • Lots of customers

Weaknesses

  • Scored midquartile or below in most of the 11 key areas surveyed, and lowest quartile in ease of implementation, governance and performance
  • Scored in the lowest quartile in overall capabilities
  • Scored in the lowest quartile for service and support

So Oracle’s strength is that lots of people buy it and its weakness is that they subsequently hate it. There’s your market leader folks!

Why did IBM drop out of the Leaders quadrant?

IBM fell out of the leaders quadrant this year, so Planning Analytics must be falling behind right? Let’s read the report to see what happened.

Strengths

  • Many improvements to enhance functionality, usablity and flexibility. In the highest quartile in three of 11 key areas
  • Used for enterprise implementations with a strong partner ecosystem
  • Improved support since last MQ

Cautions

  • Gartner is getting fewer calls about IBM than they did last year
  • Scored mid or lowest quartile in 8 of 11 key areas
  • 71% of customers recommend without reservation, but 29% recommend with some reservations

Gartner says Planning Analytics improved significantly and scored in the highest quartile in 3 categories compared to 0 categories for Oracle. So what gives? It all boils down to this: Gartner is getting fewer calls about IBM than they did last year.

That’s it. That’s why IBM is no longer a leader and Oracle is, despite outscoring Oracle in customer surveys. It’s because Gartner’s advisory fee revenue related to IBM went down.

Gartner does not evaluate software

I’m not sure what other conclusion can be drawn when the market leader fails to score above the midpoint in a single survey category.

You or I would install the software, learn the software and compare its capabilities, performance and stability relative to its peers before making a judgement. Gartner is not doing this kind of analysis. What Gartner is doing appears to be:

  • Don’t judge software yourself, send out surveys to existing software customers instead
  • Collect results
  • Ignore results
  • Rank software based on your internal consulting revenue
  • Profit!

Gartner doesn’t do a deep technical analysis of the strengths and weaknesses of software in house; they instead send out surveys and let their customers do it for them. Except they also don’t appear to take the survey results into account either based on how they’ve ranked Oracle this year.

Can you just ignore Gartner?

Unfortunately no. Despite this example of how they seem to ignore their own dubious methodology when producing these rankings the reality is that they carry incredible weight in the software industry. Executives will base decisions on Gartner’s ‘expertise’.

What you can do is read the actual report to educate yourself on the contradictions between the MQ rankings and Gartner’s own analysis and encourage others to do the same.

And above all do your own research!

Data Modules as a Single Source of Truth

by 5 Comments

The conversation spurred by my What are Cognos Data Modules blog post continues with an excellent question by Heather, who asks what to do about maintaining a single source of truth in Cognos Analytics while still using data modules. This does pose a challenge. I’ll discuss some options below and rate them using a scale of 1 – 5 Analysis Studio cubes.

Option 1: Don’t Use Data Modules

One common way to solve this problem – sadly – is to not use data modules in any capacity. Turn them off for end users and IT and stick with tried and true Framework Manager. If it aint’ broke, don’t fix it. This approach will preserve single source of the truth but at significant cost:

  • No access to new features… ever (FM is not receiving feature updates)
  • FM development cycles are very long
  • End users will have no ability to model data in Cognos…
  • Which means they won’t be able to do anything complex in Cognos…
  • Which means they’ll use Power BI instead!

Odds are if you’re reading this blog you’d prefer your users work in Cognos vs Power BI or Tableau. Choosing option 1 will definitely preserve single source of the truth but at the cost of relevance.

Final Verdict

Option 2: Treat Data Modules Exactly Like Framework Manager

Contrary to popular belief – and IBM’s marketing – data modules are not just for self-service users. The feature set of data modules has reached near parity to Framework Manager (with some notable exceptions) and has surpassed it in some very significant ways – relative time, for example. It’s also important to understand that Cognos security applies to a data module in just the same manner as a Framework Manager package.

There is in principle no reason you can’t migrate your IT modeling work to data modules wherever possible while restricting their use entirely to the IT team. In this way you can future proof your development and take advantage of a growing list of very cool features while leaving your traditional BI workflow and single source of truth intact.

However you foreclose the opportunity to take Cognos to the next level and enable real self-service.

Final Verdict

Option 3: Use Linked Tables

Linked tables allow to you use tables from one data module as the source for tables in another data module. These tables inherit all changes from the source, including field name changes, field additions/subtractions, calculation changes, SQL changes, etc…

Linked tables in action!

Using these links, it’s possible for IT to build and maintain a high quality, tested and accurate data model while still providing a significant degree of end user flexibility. In the image above, an end user has imported three linked tables over which IT has 100% control and joined them to an excel spreadsheet. To do this follow these steps:

  1. Create one or more data servers for your source databases
  2. Restrict access to create new models using those servers to IT only
  3. Build a data module that contains the tables you wish to govern
  4. Save the data module in a folder where users can access it but not overwrite it
  5. Remove the ‘break link’ capability from users. This means they cannot edit these tables in any way
  6. Teach them how to use link tables and let them go wild!

By following these steps you can ensure that all self-service data modeling for governed sources uses your logic, your calculations and even your joins – the joins you define are imported alongside your linked tables. Your users can still model to their heart’s content with their departmental or personal data but when it comes to your governed enterprise data, your rules come first.

Final Verdict

Option 4: Stop, collaborate and listen

For better or worse, speed and flexibility have decisively crushed accuracy and security in the battle for BI market share. End users routinely show up to meetings with three competing Power BI/Tableau workbooks that disagree on all relevant metrics but were produced in four days and look incredible. And people seem absolutely thrilled a this state of affairs based on recent events!

Therefore Option 4 isn’t so much a technical solution as it is a cultural one. Yes, you adopt the strategy outlined in Option 3 – but you do so cheerfully with the knowledge that your end users absolutely will colossally screw up the data in Cognos, and that it’s your job to identify and fix it when this happens. This is by far the hardest thing for long time Cognos pros to do, and it was extremely hard for me to come around to this way of thinking. The final piece of the puzzle was the following realization:

  • Users are incredible innovators in the art of producing bad data – they truly cannot be stopped
  • Trying to enforce 100% data accuracy in Cognos just drives them to other tools
  • Errors produced in those other tools are completely outside of my ability to monitor, influence or correct – as are those users
  • This cycle absolutely destroys ‘single version of truth’ for an organization as a whole, even if it is maintained for Cognos

Switching to a more open, collaborative approach in Cognos has the benefit increasing user satisfaction, organizational perception and eventually investment in Cognos. I have seen this work at my clients who have embraced it.

Final Verdict

That’s right, the final verdict for Option 4 is five PowerPlay Studio cubes of approval – the highest approval I can give.

PowerPlay was way better than Analysis Studio anyway.

Alias Shortcuts in Cognos Data Module

by 3 Comments

My recent What are Cognos Data Modules post generated some interesting discussion around the alias shortcut functionality of Framework Manager and whether or not it is available in data modules. I initially responded by saying you could make data module alias tables, however SirM challenged me that it isn’t the same at all.

SirM is right.

What is an Alias in Cognos

In my experience the primary reason to build an alias is to tightly control how Cognos executes joins. Imagine you have two tables, Sales_Fact and Time_Dim. Sales_Fact contains the fields order_date and shipped_date, both of which you wish to join to Time_Dim. You could join the tables twice and be done with it, but then you are at the mercy of Cognos as to which join it will include when generating SQL. This will introduce unintended or completely incorrect results when the ‘wrong’ join is used.

Enter Framework Manager alias shortcuts. An alias shortcut is essentially a pointer to another table that has the following properties:

  • Inherits all fields and all changes from the target table automatically
  • Ignores all relationships from the target table
  • Can be repeated as often as necessary

In our example above, we make an alias shortcut of Time_Dim called ‘Shipped_Date’ and join the alias to Sales_Fact. Shipped_Date will inherit all fields from Time_Dim but will have an independent relationship to Sales_Fact.

Alias in Data Modules

Can we do this in data modules? Let’s explore the three options we have.

Copy Tables

Copying a table does exactly what you’d expect – it makes a copy of the table in question. You can create as many copies of a tables as you want, however changes you make to the target table do not flow through to the copied table in any capacity – whether you add or remove fields, rename fields or change calculation logic.

  • Inherits all fields and all changes from target table automatically: NO
  • Ignores all relationships from the target table: YES
  • Can be repeated as often as necessary: YES

View Tables

View tables are like copy tables with a couple very important differences; views can be composites of many underlying tables and they do inherit some changes from the target automatically. Specifically, they will inherit calculation changes made to the fields in the target table but will not inherit name changes or added/removed fields without manually editing the view definition. I will do a deep dive on their usage in a future post.

  • Inherits all fields and all changes from target table automatically: SORTA?
  • Ignores all relationships from the target table: YES
  • Can be repeated as often as necessary: YES

Linked Tables

Linked tables are essentially pointers to tables built and maintained in other data modules. They are extremely useful for BI teams concerned about data quality in Cognos Analytics, as you will see below.

Linked tables inherit all properties from the target table in the model in which they were built. This means that any changes I make in the target module, whether I add or remove fields, create new calculations or edit existing logic will automatically flow through to all linked tables that reference it. This would appear to solve our alias shortcut problem, so what’s the catch?

You can only import a linked table into a data module once. It cannot be copied and any views you build on it have the view limitations outlined above.

  • Inherits all fields and all changes from target table automatically: YES
  • Ignores all relationships from the target table: YES
  • Can be repeated as often as necessary: NO

So What’s the Solution?

Whenever you find yourself reaching for the alias shortcut button in data modules, ask yourself which alias table feature is most important for the task at hand.

  • If the most important thing is automated inheritance of all future changes, build a link table
  • If the most important thing is re-using the same table over and over, build a view table
  • In most instances, do not build a copy table

In reality, a view table should fit your needs in most circumstances. Yes, you will need to manually intervene to inherit certain changes listed above, however this process takes about 30 seconds per table. Not ideal but something most of us can live with.

How to Proceed

Alias tables have historically filled a very important role in building large scale Cognos models in Framework Manager, and their absence in data modules creates challenges to modeling the way we have for over a decade. Certainly many of my most skilled customers feel that without this functionality data modules don’t have much use.

It’s worth considering then how it is that Tableau and Power BI managed to dominate the mid-late 2010s BI market with a total absence of an equivalent to alias shortcuts or many other ‘enterprise’ BI modeling features? The answer lies in the culture and practice that these solutions enabled which delivered faster results for business users. How to apply these practices to Cognos using data modules will be a major theme of this blog going forward.

What are Cognos Data Modules?

by 9 Comments

Cognos Data Modules are a web-based data acquisition, blending and modeling feature available in Cognos Analytics. They first hit the scene as part of Cognos 11 and are meant to supplement and eventually replace Framework Manager for both self-service and IT data modeling needs. I’ll pause for a second to let you long-time Cognoids hyperventilate a little… is everyone back? Good. Through this and subsequent posts I’ll try to dispel misconceptions about this awesome feature of Cognos while making you comfortable and – dare I say – excited to use them.

Cognos data module in action
Data modules – the wave of the future

Data Module Features

Imagine a data modeling solution that has the following features:

  • Easy to install and manage
  • Join dozens or hundreds of tables across multiple databases
  • Execute cross-grain fact queries
  • Build simple or complex calculations and filters
  • Build alias, view, union and join virtual tables
  • Secure tables by groups, roles and data elements
  • Create OLAP-like dimensional hierarchies
  • Enterprise governance, auditability and security

 ‘Okay easy, I’m imagining Framework Manager’ you’re thinking right now. Yes! But, add in:

  • Natural-language and AI powered auto-modeling
  • Automatic join detection
  • Easy integration of excel data
  • Automatic extraction of year, month, day from date data types
  • Automatic creation of relative time filters (YTD, MTD, etc..)
  • Automatic creation of relative time measures (YTD Actuals, MTD Actuals, etc…)
  • In-memory materialized views
  • In-memory query cache

‘Well that’s not Framework Manager… it must be Tableau, right!?’ No, in fact Tableau doesn’t offer even half of these capabilities. This is what every Cognos Analytics customer gets out-of-the-box in data modules today, with more features being added all the time.

Who are Data Modules for?

Many of my longtime customers have the misconception that data modules are for ‘end users’ only and that real data modeling can only be accomplished in Framework Manager. Conversely my new customers have built entire BI practices while having no idea what Framework Manager is. Clearly something is out of sync here, so let me make it very clear: Who are data modules for? If you’re reading this, the answer is you.

The Business User

The line between ‘end users’ and the BI team has gotten fuzzy in the last few years as increasingly complex models are built by people outside the IT department. Data modules are ideal for someone who wants to quickly and easily combine enterprise data with departmental data or excel spreadsheets and cannot wait for IT to build an FM package or SSAS cube. The interface is clean and easy to use and the ease of creating custom groups and building relative time calcs makes data modules an ideal place to combine data – even easier than Excel in many cases. As an added bonus, it’s very simple for the IT team to take a ‘self-service’ data module and incorporate into enterprise reporting without significant development work.

The Cognos Pro

Many Cognos pros kicked the tires in 2016 and could only see the yawning chasm of functionality that separated data modules from Framework Manager, myself included – for years I encouraged my clients to consider them for niche applications but to rely on FM for anything important or difficult. No longer! As of the 11.1 release, data modules have reached feature parity with Framework Manager is almost all respects and even surpassed FM in important modeling automation tasks like relative time automation. It is no longer the obvious choice to default to Framework Manager for new Cognos development.

Data Modules vs Framework Manager

Given the enhancements to data modules, which should you choose? As of the 11.1 release my recommendation is to do all new development in data modules for the following reasons:

  • Significantly easier and faster to create
  • Great features like relative time, date column splitting, grouping
  • Target of all future development
  • Unlock modern BI workflow

These points will be explored in a follow up blog post – for now I’ll leave you with a final thought. My new clients use the same ol’ Cognos to deliver with the speed and scale you’d expect from Tableau or Power BI – my friend Vijay can tell you all about it. The key differentiation between them and legacy Cognos installations with orders of magnitude more resources is the embrace of data modules and the iterative, build-it-in-prod approach to BI delivery that data modules enable.