The Blog

Framework Manager vs Data Modules

November 20, 2019 by Ryan Dolley 3 Comments

I have spent a lot of time showcasing data modules to audiences across the world in-person and in livestreams, helping people understand how and why to use them as part of Cognos Analytics. The most consistent question I receive – by far – is about understanding framework manager vs data modules. There are a lot of outdated opinions and outright misconceptions floating around so let me outline the exact feature differences between framework manager and data modules as of Cognos 11.1.4.

What do data modules and framework manager have in common?

The answer is ‘a lot’ but this wasn’t always the case. In the 11.0 releases data modules were missing many essential framework manager features and didn’t offer compelling reasons to switch. Of course that has changed. As of 11.1.4 framework manager and data modules both:

Produce data models that can be used with all Cognos 11 features
Join dozens or hundreds of tables across multiple databases
Execute cross-grain fact queries (aka the dreaded determinants)
Build simple or complex calculations and filters
Build alias, view, union and join virtual tables
Secure data by groups, roles and users
Create OLAP-like dimensional hierarchies
Offer enterprise governance, auditablity and security

Oftentimes people washed their hands of data modules a couple years ago and are surprised to see virtual tables, cross-grain fact queries and security by groups. These features may exist in both but the implementation in data modules is superior from a usability perspective.

Column dependencies can handle degenerate dimensions, unlike framework manager — Column dependencies go beyond what was possible using determinants in FM

What do data modules offer that framework manager does not?

Again, the answer is ‘a lot’. The 11.1. release takes data modules beyond what is possible in FM with a host of powerful capabilities and quality of life enhancements. The following features are either exclusive to data modules or done infinitely better in data modules.

Natural-language and Ai powered auto-modeling
Automatic join detection
Easy integration of excel data
Ability to easily clean data
Flexible hierarchies that go up, down and across (navigation paths)
Easy measure binning and attribute grouping
Easy extraction of year, month, day, etc… from data data types (split)
Automatic creation of relative time filters (YTD, MTD, PYMTD, etc…)
Automatic creation of relative time measures (YTD actuals, PYTD actuals, etc…)
In-memory materialized views within Cognos Analytics
In-memory query cache
Easy multi-model inheritance for single source of truth
Degenerate dimension aggregation (column dependencies)

Some of these features are absolute game changers for how I craft highly performant, easy to use and self-service friendly data models. Consider ~~the coconut~~ relative time; because this was such a titanic brain buster in framework manager only the most skilled developers could deliver. Now it takes minutes for end users to implement.

Building a dozen relative time filters can be done in as little as five clicks. — It took five clicks to build the relative time filters that take ~1 trillion years in FM

What are data modules missing?

There are still some things data modules lack:

Object level security
DMR capabilities
Parameter maps
Multiple connections for data servers

If I’m being honest, I don’t really recommend you use many of these features for new development in 2019 unless you absolutely have to, particularly DMRs. DMRs are very powerful for those who know MDX but a true maintenance and self-service nightmare in the long run. I cannot count the number of clients who are stranded with critical DMR based reports they cannot understand. In any case, a little bird told me that DMR-like functionality will grace data modules soon.

Going beyond the feature list

Comparing framework manager vs data modules feature for feature, we can see how data modules have few shortcomings and offer huge advantages. While this is a common way for IT folks to think (and I would know, I’m one of them!), I argue that it badly misses the point. By using data modules an IT professional can do weeks of FM work in an afternoon while a self-service user can easily accomplish tasks that will otherwise be done in Power BI. I repeat it often but I’ll say it again – data modules are the key to modernizing your Cognos Analytics environment and delivering content with the speed modern users demand.

What do you recommend?

I’ll parrot Cognos offering manager Jason Tavoularis and say, ‘use data modules unless you can’t.’ And as you can see above, the list of reasons you can’t has become quite short. I start ALL consulting engagements under the assumption that we’ll be building data modules and I’m always happy with the results.

Cognos Analytics Forecasting Overview

November 7, 2019 by Ryan Dolley Leave a Comment

Cognos Analytics 11.1.4 brings exciting and long sought forecasting capabilities to the platform. While Cognos is far from the first analytics tool to offer forecasting, IBM’s implementation is more flexible, more powerful and easier to use than competitors. Let’s take a dive into Cognos Analytics forecasting to understand why.

Cognos Analytics forecasting is a powerful and easy to use forecasting tool — Cognos Analytics forecasting is just really cool.

Where is forecasting available?

Forecasting is available in line, column and bar visualizations in Dashboards, Stories and Explorations as of 11.1.4. Report Authoring is on the horizon (in fact there’s a ton of cool stuff coming for Report Authoring.) The ability to forecast column and bar charts sets Cognos apart from Tableau and Power BI. Cognos hides the forecasting button until a visualization meets the following criteria:

Visualization is the proper type (line, column or bar)
Visualization has a measure in the Y axis
Visualization has a time element in the X axis

Once this criteria is satisfied the forecast button appears next to the insights button in the top right of the visualization.

You start from any line, column or bar chart — Forecasting is available for line, column and bar charts

In the above example I use the ‘Great outdoors data module’ available in the samples. Using the Sales table I apply Revenue to the Y axis and nest Year and Month on the X axis – the perfect scenario for forecasting.

Single category forecasts

One click of the forecast button is all it takes to build a predictive forecast — Forecasting a single category is simple

Click the forecast button, turn the feature on and there you have it – your very own Cognos Analytics forecast! You may immediately notice that the slope of the line changed dramatically once a forecast was applied. I first assumed this was a bug until I clicked the yellow ‘!’ next to the forecast button and read these magical words:

Cognos will even automatically re-order your time categories — Forecasting automatically orders time categories chronologically!

That’s right – Cognos automatically reordered the time categories to be in chronological order rather than alphanumeric! I’ve been waiting for this feature for all my life and it’s finally here. Now we just need IBM to automatically apply it to all visualizations, not just forecasts.

Multi-category forecasts

Adding additional categories builds a multi line or bar forecast — Forecasting multiple categories, just as easy

In the above example I add ‘Retailer Type’ from the Retailers table to the color attribute of the visualization. Cognos instantly re-applies the forecast to each individual category. This is one advantage Cognos has on Power BI, which currently forecasts a single category only. This makes data discovery on forecasts extremely fast and easy.

Interacting with forecasts

Hovering over an individual point in the forecast will show you the confidence upper bound, forecast and lower bound for that intersection of the chart:

In this example I hover over the projection for ‘Outdoors Shop’ in September, 2018. You can see Cognos’ predictions for the upper and lower bound.

Focusing on a single category will visualization the confidence interval — You can visually see the upper and lower bound as well

Clicking on the line for ‘Outdoors Shop’ will filter the dashboard by this category, bring the selected category into visual focus and plot the confidence intervals as they evolve in the forecast. Super duper cool.

Configuring forecasts

Forecast configuration options are available in the menu that appears when the forecast button is clicked.

Cognos analytics forecasts have several configuration options to customize how the forecast is generated — Forecast configuration options help tailor results

Let’s take a look at what these options do:

Forecast periods: The number of periods included in the forecast. The default ‘Auto’ will project forward 20%. If you have 10 months of data, auto will generate an additional 2 months of forecast
Ignore last periods: Ignore last periods is useful for cases where you have incomplete data at the end of your chart – for example, the month of November currently has 6 days worth of actuals. Ignoring this incomplete period creates a more accurate forecast.
Confidence level: Controls the confidence level displayed on the tool tip and confidence visualization – options are 90%, 95% and 99%.
Seasonality: Cognos automatically detects seasonal fluctuations in your data and accounts for them in its forecasts – think about retailer revenue during the holidays for example. The default ‘Auto’ setting will build multiple models with different seasonal periods and and select the best one, however you can specify a season period by entering numbers here.

That last option is incredibly powerful and another strong advantage for Cognos vs Power BI and Tableau. The inability to detect and account for seasonal variation when forecasting renders this functionality useless for many industries – retail, hospitality, utilities – and only Cognos has it.

Understanding forecasts

Cognos uses exponential smoothing models to generate these forecasts. I don’t know what the means either but you can read about it here. The important thing to know is that you have access to the forecasting statistical details in the data tray.

Cognos analytics forecast statistical details are available for each category in the visualization — For those that know, here are the statistical details for the forecast

I’m a BI/DW guy and this screen might as well be magical incantations to me, but the fact that Cognos provides this level of detail means that I can always find a friendly wizard to explain it. IBM has a nice explanation that I also don’t understand here.

Bar chart forecasts

Switching to a column/bar chart retains the forecast and forecast parameters that were applied to your line chart.

One thing it doesn’t do, which I just discovered in writing this blog post, is automatically sort the time category in chronological order like it does with line charts. I’m not sure if this is purposeful, an oversight or a bug. You can vote for my feature request to rectify this here.

Cognos Analytics forecasting is the real deal

I have to say, I think this feature is extremely well done and I’m certain end users are going to love it. If you need help prepping your environment for self-service see my guide here. Allowing your users to build their own data modules will make forecasting even more powerful, read more about it here.

IBM was late to the market with it but delivered something much better than their competitors. This has been a recurring story in the last year – Cognos has come so far since the 11.0 releases. It’s up to IBM – and to us – to make sure the broader BI community is aware of the good things going on in Cognos Analytics.

Cognos Analytics Linked Modules

October 29, 2019 by Ryan Dolley Leave a Comment

If you’ve been following this blog you know that I love Cognos data modules to death. They are easy to learn, easy to use and full of functionality. Today we’re going to explore using Cognos Analytics Linked Modules to maintain enterprise data governance while enabling self-service. This entry is meant to give a straightforward implementation guide to reinforce concepts outlined in the posts Data Modules as a Single Source of Truth and How to Organize Cognos for Self-Service.

Companies do not have to choose between governance and self-service — Like Jedi vs Sith… can you truly have both?

How and why of linked modules

Using linked modules is a slam dunk way to enable self-service analytics on a foundation of trusted and governed data. When you link two modules you are a creating a relationship in which:

The IT expert can build and maintain a complex module with high quality data that requires significant skill to create
The power user can utilize high quality data as a foundation for analysis but customize and extend at their own pace, not IT’s
The casual user can build dashboards and run reports that meet their needs without waiting for IT

This works so well because the modules of the power user automatically inherit all the complex relationships, transformations and logic from the enterprise module prepared by IT in near real-time without having to do anything. It’s magic.

Step 1: Build an enterprise data module

The enterprise module is typically – but not always – built by IT, more often than not the result of a much longer process of data warehouse design and ETL. Regardless of how its created or what it contains the enterprise module will server as our example though any two modules can be linked together.

Any data module can serve as a source for another data module — The enterprise module serves as the source / parent when linking data modules

Our example uses ‘Great outdoors data module’ from the Cognos samples. This module is located in the data folder. Once built the enterprise module resides in a folder where self-service users can easily access it. I advocate creating a data library folder for this purpose.

Step 2: Use a data module as a source

IT has built and deployed an enterprise module with validated and accurate data, which takes a long time. Now a power user needs to extend that module with some additional context and can’t wait for IT to do it. The solution is simple: ~~use Tableau~~ Build a new data module using the enterprise module as a source!

Adding a data module source is no different from adding any other source — Adding a data module source is extremely easy, fun and profitable

When creating a new data module, users can select other modules as sources in the ‘select sources’ screen. In our example, select ‘Great outdoors data module’ and hit ‘OK’. That’s it. We’ve built a linked module.

Data module sources appear in teal and have a link icon — A linked module has visual cues to let you know that it is indeed a link

The tables above appear in teal rather than blue and have a link icon, indicating that they are actually inherited from another module. At this point any changes made to the tables of ‘Great outdoors data module’ will automatically flow through to this module with absolutely no intervention.

Step 3: Extend the enterprise module

You’ll notice that the teal linked tables do not offer much in the way customization by design. They inherit everything from the parent module. This means users cannot change what comes from the enterprise module but they can extend it with additional logic and data.

Linked tables don't have all the options of normal tables as you're trying to preserve governance. — No options = no opportunity to mess with governed data

Cognos restricts us to creating data groups, navigation paths and editing some properties for product line – a far cry from what is available in the source module. However we still have considerable power to extend the enterprise module. Let’s look at some examples.

Creating calculations

In the example above the calculated field ‘Revenue Variance Amount’ contains the fields ‘Revenue’ and ‘Planned Revenue’ from the sales table. This custom calculation is built entirely using elements inherited from our enterprise module. At many of my customers such a simple request spends weeks or months in the IT backlog but here we accomplish it in minutes while still using centrally modeled and governed data. Yowza!

Adding additional data

Here I join a Cognos data set called ‘Sales Staff Analysis’ to the tables from the enterprise module via two conjoined dimensions. Joining a spreadsheet is just as easy, which is what the vast majority of power users want to do.

These are just two examples of extending an enterprise module using the linked module functionality. Users can do almost everything data modules have to offer while still relying on trusted data, including building table views and aliases, creating drill paths, custom grouping and filter, row level security, etc…

Step 4: Watch the linking magic in action

Before the module can be used it must be saved. Users typically save modules to ‘My content’ but it is critically important that you allow them to save self-service modules to a location where they can easily share with coworkers. Now let’s close this module and test our module link.

Almost all changes made to the source module flow through

Here I create a new calculation in our enterprise module called ‘Revenue Percent Variance’ and click ‘Save’. I then immediately create a dashboard using the self-service module as a data source. Keep in mind that I made absolutely no updates to the self-service module before doing so.

The changes flow through just like magic!

As you can see, ‘Revenue Percent Variance’ is immediately available for use in Dashboards even when they reference the self-service module rather than the enterprise module.

Putting it all together

Allow me to summarize what we just did:

An enterprise module is made available as a data source
Power users extend the enterprise modules with their own logic and data in self-service modules
Content is built using both the enterprise and self-service modules as a data source
Updates to the enterprise module flow through to all content regardless of which data source was used

Single source of truth meets self-service

Nobody but Cognos offers this combination of data governance and self-service. It’s a game changer for organizations who truly embrace it. Book some time with me to explore these ideas further (there’s a link on the screen) and be sure to check out my live impressions of the recent 11.1.4 release.

How to Organize Cognos for self-service

October 28, 2019 by Ryan Dolley 1 Comment

Most Cognos environments are organized to support a world of IT authored report consumption – a fact that I confirm over and over as I present about Cognos modernization across the United States and am asked by beleaguered but hopeful looking Cognos admins, ‘How am I supposed to support all this self-service stuff?’ Never fear – here’s my take on how to organize Cognos.

Why your environment needs a re-org

11.1.4 is out, is excellent and you need to utilize all its features in production ASAP. Reticent clients sometimes object that their existing folder structure and security cannot support all the new features. Honestly they probably can’t. The typical IT managed legacy Cognos environment has some combination of the following:

High level folders that reflect the org structure (finance, HR, etc…) in which users can do nothing
Low level subject area folders nested within the org folders in which users can do nothing
A hidden data sources folder that users can’t find
A self-service folder where old timers use Query Studio
Non-prod environments where all real work must be done
A deployment cycle that takes weeks to bring finished reports to production

No path leads to modernization with these roadblocks to user adoption in place. Feel no shame, however, if this describes your environment in part or whole – this was absolutely best practice when most Cognos environments were first established.

Towards a new folder structure

Modern Cognos environments deliver consumer oriented experiences built around use cases rather than org structures or IT priorities. Redesigning folder structures and security to achieve this feels daunting – but it’s crucial. Here’s an example of what it looks like:

A modern Cognos Team Content folder structure

Curated Content

Curated content is the closest thing to a traditional IT-centric approach and is where a large portion of existing IT authored reports wind up. Mission critical reports live here alongside anything too high profile to mess up. Tight security and IT control are the name of the game here.

Nobody saves to curated content without IT approval and strict documentation. If your requirements involve audit, regulatory reporting or the CEO’s eyeballs it goes here.

Data Library

The data library is a semi-curated area and is THE PLACE to go to find data in your company. The data must be high quality and IT should exercise significant oversight, however data stewards and trusted power users have the ability to promote data to this location.

Do not structure the data library by data source or modeling tool – that’s thinking like IT! It’s okay to have folders mingled with packages and data modules. Using linked modules to reference the data library maintains single source of truth.

Sensitive data targeted at only a specific department – HR data for example – can be located here as well provided you have appropriate object or data level security in place but may be best kept in department content

Department Content

Department content is a semi-curated area controlled by the data stewards and power users in each of your business units. Much of your existing content migrates here and becomes the responsibility of the business unit to secure, maintain and update. This means granting admin, modeling and report authoring powers to trusted individuals in the business.

Organize cognos content by department — Data stewards and power users maintain department content – not IT!

Yes this means dissolving the responsibility for maintaining department content folder security to the business. This is a shocking recommendation, I know, but this power belongs with those who know the data and user community best.

Help

Help contains everything to assist your users with Cognos. The Cognos samples go here. Example content you build goes here. Everyone has access to this location.

Self-Service

Self-service is hands off to IT. Anybody with access to Cognos can fully utilize all their capabilities in this location without asking permission first. That means making folders, building data modules, uploading excel files, etc. I mean it – everything!

You are indeed inviting chaos into Cognos, but this chaos already exists in your organization – it goes by the names ‘Tableau’ and ‘Power BI’. Giving full access to all Cognos features in the self-service folder encourages users to, well, use Cognos.

This is not to say you surrender all control – rather, you closely monitor what users are doing using the audit reports or a product like PMsquare’s Thrive and intervene as necessary under the following conditions:

Users are sharing inappropriate data
Users need help utilizing Cognos
Self-service content gains too wide an audience

Pmsquare Thrive will help — Thrive will tell you what they are doing…

The beauty of the self-service folder is this: When a user builds a dashboard based on spreadsheets that winds up being used by 200 people, you’ll know. And you and the relevant data steward from the business can transition that report to department reporting or curated content where it belongs.

Now you’re ready to organize Cognos

I hope this gives you some inspiration to deploy a more modern approach to your Cognos environment. There is much more to this than simply changing the folder structure, but a new folder structure is a crucial piece of the puzzle and one that I get asked about all the time. You’ll also need to consider things like persona-driven development, data governance with linked modules and thinking like an app instead of a report. I’ll cover all these topics and more going forward.

Cognos 11.1.4 Performance Tips

October 17, 2019 by Ryan Dolley Leave a Comment

Are you attending Data and AI Forum in Miami next week? Want to talk some Cognos with me? Snag a half hour of my time to chat free from sales guys.

Thank you to those who attended Tuesday’s 11.1.4 livestream! Technical issues aside I had a great time doing it and will be hosting more in the future. There were some comments about performance in recent releases that I promised to address but didn’t – so here are my Cognos 11.1.4 performance tips!

Tip one: Dispatcher > Gateway

Cognos 11 brought with it the option to dispense with the gateway entirely and sending your traffic directly to the dispatcher. This is, among other things, a performance hack that can pay big dividends, particularly if your Cognos environment takes an extremely long time to load or if the UI feels generally sluggish.

The downside here is that for most enterprise deployments relying on a single dispatcher is just not an option for a ton of reasons – including little things like SSO, load balancing and disaster recovery. But if you are able to try this you might find it helps quite a bit.

Tip Two: Reality > Minimum Requirements

The minimum requirements for Cognos 11.1.4 are currently listed at two CPUs. This is a straight up no-go for using Cognos in almost all circumstances regardless of what version you’re on so long as that version starts with the number 11.

A tale of two configs

I typically deploy a single-instance Cognos machine on AWS with the following specs and have no performance issues for demos or small deployments:

ec2 t3.xlarge
4 vCPUs @ 2.5Ghz
16GB RAM

With 11.1.4 this configuration suffers from a laggy UI and poor query performance, especially in Dashboards. I observed CPU utilization hitting 100% frequently. As a remedy I migrated my AMI to:

ec2 c4.4xlarge
16 vCPUs @ 3.4Ghz
32GB RAM

This adjustment resulted in extremely snappy performance throughout the UI and like-for-like dashboard query execution improved from ~25 seconds to ~2 seconds.

Why did this jump so much?

Cognos 11.1.4 is doing A LOT more than 11.0.x or 10.x ever dreamed of with a host of new interactive services running in the background – services that do things like natural language query, automatic visualization or dashboard creation, on-the-fly predictive forecasting, integrated data prep and modeling…

Tip Three: SSD > HDD

One often unexplored area of performance improvement for Cognos 11.1.4 (or anyone using Transformer for that matter) is increasing the speed of your HDD or migrating to an SSD. All of the new features rely on either the Cognos 10 DQM engine or the Cognos 11.1 ‘Flint’ spark-sql engine to process queries. In either case data is written to and read from disk so IO becomes larger factor in determining performance than ever before, especially for any interactive task.

Tip Four: Your Browser Matters Now

Go ahead and conduct an experiment for me – pop into Dashboards on your local machine, build a bunch of visualizations and observer your resource use. You may notice that Cognos running in Chrome is capable of maxing the CPU on your laptop. It is capable of maxing the CPU on my monster gaming desktop as well if I push it hard enough.

So the final place I’d look to juice performance is actually at the hardware your users are toting around with them. It may seem a little unfair but just imagine life as a poor Power BI or Tableau administrator – outdated and slow laptops define the entirety of their user experience.

Cognos 11.1.4 Performance Summary

All of this is to say that I have had success increasing performance by significantly increasing the specs of the application server. This isn’t necessarily shocking but it’s comforting in a way because it means the problem isn’t bad code. It also means that if you’ve been skating by on the same server config you put into place in 2015 it’s probably time for an upgrade.

Join me at Data and AI Forum 2019

October 11, 2019 by Ryan Dolley Leave a Comment

I’ll be attending Data and AI Forum again this year, as I do every year. Data and AI Forum is THE conference to attend for Cognos, Planning Analytics and other IBM analytics technologies. THINK is the big IBM conference but for us, this is the valuable one.

If you’re interested in saying hello you can find me at the PMsquare booth or attend one of my sessions:

If you’re struggling to modernize your Cognos environment, the second session is a treat.