Cognos Data Modules are a web-based data acquisition, blending and modeling feature available in Cognos Analytics. They first hit the scene as part of Cognos 11 and are meant to supplement and eventually replace Framework Manager for both self-service and IT data modeling needs. I’ll pause for a second to let you long-time Cognoids hyperventilate a little… is everyone back? Good. Through this and subsequent posts I’ll try to dispel misconceptions about this awesome feature of Cognos while making you comfortable and – dare I say – excited to use them.
Data Module Features
Imagine a data modeling solution that has the following features:
- Easy to install and manage
- Join dozens or hundreds of tables across multiple databases
- Execute cross-grain fact queries
- Build simple or complex calculations and filters
- Build alias, view, union and join virtual tables
- Secure tables by groups, roles and data elements
- Create OLAP-like dimensional hierarchies
- Enterprise governance, auditability and security
‘Okay easy, I’m imagining Framework Manager’ you’re thinking right now. Yes! But, add in:
- Natural-language and AI powered auto-modeling
- Automatic join detection
- Easy integration of excel data
- Automatic extraction of year, month, day from date data types
- Automatic creation of relative time filters (YTD, MTD, etc..) and measures (YTD Actuals, MTC Actuals, etc…)
- In-memory materialized views (data sets)
- In-memory query cache
- Direct access to members for relational sources!
‘Well that’s not Framework Manager… it must be Tableau, right!?’ No, in fact Tableau doesn’t offer even half of these capabilities. This is what every Cognos Analytics customer gets out-of-the-box in data modules today, with more features being added all the time.
Who are Data Modules for?
Many of my longtime customers have the misconception that data modules are for ‘end users’ only and that real data modeling can only be accomplished in Framework Manager. Conversely my new customers have built entire BI practices while having no idea what Framework Manager is. Clearly something is out of sync here, so let me make it very clear: Who are data modules for? If you’re reading this, the answer is you.
The Business User
The line between ‘end users’ and the BI team has gotten fuzzy in the last few years as increasingly complex models are built by people outside the IT department. Data modules are ideal for someone who wants to quickly and easily combine enterprise data with departmental data or excel spreadsheets and cannot wait for IT to build an FM package or SSAS cube. The interface is clean and easy to use and the ease of creating custom groups and building relative time calcs makes data modules an ideal place to combine data – even easier than Excel in many cases. As an added bonus, it’s very simple for the IT team to take a ‘self-service’ data module and incorporate into enterprise reporting without significant development work.
The Cognos Pro
Many Cognos pros kicked the tires in 2016 and could only see the yawning chasm of functionality that separated data modules from Framework Manager, myself included – for years I encouraged my clients to consider them for niche applications but to rely on FM for anything important or difficult. No longer! As of the 11.1 release, data modules have reached feature parity with Framework Manager is almost all respects and even surpassed FM in important modeling automation tasks like relative time automation. It is no longer the obvious choice to default to Framework Manager for new Cognos development.
Data Modules vs Framework Manager
Given the enhancements to data modules, which should you choose? As of the 11.1 release my recommendation is to do all new development in data modules for the following reasons:
- Significantly easier and faster to create
- Great features like relative time, date column splitting, grouping
- Target of all future development
- Unlock modern BI workflow
These points are explored in detail here – for now I’ll leave you with a final thought. My new clients use the same ol’ Cognos to deliver with the speed and scale you’d expect from Tableau or Power BI – my friend Vijay can tell you all about it. The key differentiation between them and legacy Cognos installations with orders of magnitude more resources is the embrace of data modules and the iterative, build-it-in-prod approach to BI delivery that data modules enable.
Have they addressed the issue of being unable to alias tables? We heavily rely on that as there are many time keys on say an invoice or order, there’s the order date, ship date, delivery date. We alias our time table multiple times to support the multiple foreign-key linkages between the order/invoice detail table and the time table.
Yes, you can now duplicate tables to do exactly what you describe. They don’t use the term ‘alias’ for it but it’s functionally the same.
That’s not the same functionality . If you create 10 copies of the date dimension table, then maintenance will take 10 times.
Yes, you’re right. See here: https://ibmblueview.com/data-module-alias/
What do you do about the multiple sources of truth? – say two users create two different data modules and add fields with calculations, but each says their information is correct and their numbers when comparing reports don’t match. This type of thing often causes the BI team to have to go back and figure out why… It seems like Framework manager creates that ‘one source of the truth’ so that there’s not a lot of different packages (now modules) floating around. How do you keep governance intact while still using modules?
Heather, great question. So great that I wrote a blog post about it. https://ibmblueview.com/data-modules-as-a-single-source-of-truth/
Thank you for your input!
Can you do Data Security in data modules yet? In FM I can limit the data that a user can see based their attributes in Active Directory…I haven’t yet found a way to do the same thing in data modules. But if you can point me in the right direction that would be great.
Yes, you can – not to quite the same degree as in FM but they’re working towards that. I’ll write a longer article about this at some point, here is the IBM documentation
One thing to note – this applies to the data server, not an individual module. If you have multiple modules that require independent, conflicting security profiles built on the same data server… well, I think your only option at this point is to create multiple servers.
I have a customer who is very keen on using the lineage function – does this work on data modules in the latest release?
Ryan, do you have any knowledge of lineage availability in data modules. For me, this would be a significant loss compared to FM.
Paul, unfortunately there are no lineage capabilities within Cognos for data modules at this time. You can do an impact analysis and see downstream content from any data source, including a data module, in the Thrive software that PMsquare makes and that I manage. I would also expect this functionality to arrive within the next year.
Hello Ryan, thanks fo your excellent articles !
One issue we are meeting with data modules compare to FM, is that you have to choose a schema when loading tables (by design). You do not have to do that in FM and you can let the database (Oracle for us) choose the schema depending on the connection used (several connection for the same datasource/dataserver). It allows us to dispatch user access to several identical schema’s data (one schema by region for a country, one server by country) with only one pack.
More : with parameter map (reading an Oracle table) and some FM function, we are even able to dynamically “forced” in FM the schema to use depending on the user or even throw a prompt to ask wich country they want to see.
Do you know if functionnalities that can help switch dynamicaly between schema for a datamodule are under consideration (I posted some) ?
I found a workaround thanks to rteruyas here : https://www.cognoise.com/index.php?topic=33317.0 but it is a bit harsh to do at a large scale …
I read your article on Relative Time in FM, and using Data modules to bring in relative time filters. Then I read this one. My team is exploring Data Modules and are building out one currently. I am wondering, can we use any date table for Relative Time or do we need to use the samples? Our Fiscal Year starts on 12/28 and ends on 12/27, each monthly period starts on the 28th. I wasn’t sure if any of the samples would meet our needs. Thanks!
Hi Ryan,
Is it possible to implement dynamic data security in data modules 11.1.5? We need to have RLS on multi dimensions with an either or kind of a lookup. Have tried to detail out with example below.
user_master
userid username
1 alex
2 jane
3 mont
4 kale
user_access
user id agent id product id office id
1 all all l10
2 p101 all all
3 all z80 l30
4 all z90 all
fact_revenue
agent id product id office id revenue
p101 z80 l10 1000
p101 z90 l30 2000
p103 z80 l30 4000
p103 z80 l10 3000
p104 z90 l10 1500
p105 z80 l10 2500
The results should be as below
user id revenue
1 8000
2 3000
3 4000
4 6000
It would be really helpful if dynamic data security can be used to accommodate the above scenarios.
Is there an alternative solution to implement Parameter Maps using Data Modules?
I’m new to modules. I’m trying to blend 2 Cognos packages in a module. Any helpful videos which will show this process.
Thanks,
Joseph Bankston
I’m making a youtube video to answer this question, check out youtube.com/pmsquare to see it soon.
Hi Ryan, Thanks for the article.
Are there any limitations for using Data Modules when it comes to the size of data set results? We have a business client who built a Data Module based on a fact table that is performing very poorly. The fact table contains over 35 million rows of records.
This causes very slow performance for the reports built on top of this DM.
What’s your recommendation for cases like this?
– Date Sets build from DM and refreshed regularly?
– Limit DM data results with various filters?
I look forward to your comments
Martin, – apologies for the very slow response to your question, I just noticed it!
My first thought is that 35M rows is too many to put in a data set. They top out around 8 – 10M rows. Is this data available via an FM package, and is the performance similar? I would not expect a big delta between FM and DM, so if you’re experiencing that we should bother IBM about it. Otherwise I think you have two options:
1. Build a data set that contains only the data needed for analysis that tops out around 8M rows. Make sure to sort it based on the values likely to be used in filters for best performance.
2. Figure out how to make the database process the query faster. 35M is not that many rows depending on your DB technology and tuning.
I realize you’ve probably already solved this problem. If you see this, let us know what happened.
Data modules would be useful if they were not so useless.
Lack of multilingual and parameterization support in real world applications will make you abandon DM before you even start the project.
https://www.ibm.com/docs/en/cognos-analytics/11.2.0?topic=dmfm-framework-manager-features-not-supported-by-data-modules
Ryan, is there a limit to the number of tables you can bring into a DM?
So there isn’t a hard limit, but after a couple hundred performance gets really really poor.