Why Gartner is Worthless

Gartner is probably your organization’s go-to resource for assessing software which is a shame because Gartner is worthless. Their process is misunderstood by the industry and their magic quadrants are utterly misleading. It’s all the worse for their ubiquity as a key reference for decision makers. Why do I feel this way? Let’s examine the recent Magic Quadrant for financial planning and analysis as a case study.

When a picture is worth no words

Let’s be honest – almost nobody reads these Gartner reports. They look at the picture and they pick the top 3-5 for an RFP figuring they have their bases covered. So looking above, I’m going to bring in Oracle, Workday and Anaplan for my new planning solution and I’m going to be biased towards Oracle as being the ‘market leader’.

What makes Oracle a market leader?

Let’s look past the MQ image though, scroll down through the report and read what they actually have to say about Oracle – the market leader! I’ll summarize below.

Strengths

• Lots of customers
• High brand recognition
• Lots of customers

Weaknesses

• Scored midquartile or below in most of the 11 key areas surveyed, and lowest quartile in ease of implementation, governance and performance
• Scored in the lowest quartile in overall capabilities
• Scored in the lowest quartile for service and support

So Oracle’s strength is that lots of people buy it and its weakness is that they subsequently hate it. There’s your market leader folks!

Why did IBM drop out of the Leaders quadrant?

IBM fell out of the leaders quadrant this year, so Planning Analytics must be falling behind right? Let’s read the report to see what happened.

Strengths

• Many improvements to enhance functionality, usablity and flexibility. In the highest quartile in three of 11 key areas
• Used for enterprise implementations with a strong partner ecosystem
• Improved support since last MQ

Cautions

• Gartner is getting fewer calls about IBM than they did last year
• Scored mid or lowest quartile in 8 of 11 key areas
• 71% of customers recommend without reservation, but 29% recommend with some reservations

Gartner says Planning Analytics improved significantly and scored in the highest quartile in 3 categories compared to 0 categories for Oracle. So what gives? It all boils down to this: Gartner is getting fewer calls about IBM than they did last year.

That’s it. That’s why IBM is no longer a leader and Oracle is, despite outscoring Oracle in customer surveys. It’s because Gartner’s advisory fee revenue related to IBM went down.

Gartner does not evaluate software

I’m not sure what other conclusion can be drawn when the market leader fails to score above the midpoint in a single survey category.

You or I would install the software, learn the software and compare its capabilities, performance and stability relative to its peers before making a judgement. Gartner is not doing this kind of analysis. What Gartner is doing appears to be:

• Don’t judge software yourself, send out surveys to existing software customers instead
• Collect results
• Ignore results
• Rank software based on your internal consulting revenue
• Profit!

Gartner doesn’t do a deep technical analysis of the strengths and weaknesses of software in house; they instead send out surveys and let their customers do it for them. Except they also don’t appear to take the survey results into account either based on how they’ve ranked Oracle this year.

Can you just ignore Gartner?

Unfortunately no. Despite this example of how they seem to ignore their own dubious methodology when producing these rankings the reality is that they carry incredible weight in the software industry. Executives will base decisions on Gartner’s ‘expertise’.

What you can do is read the actual report to educate yourself on the contradictions between the MQ rankings and Gartner’s own analysis and encourage others to do the same.

And above all do your own research!