Performance Dashboards
Measuring, Monitoring, and Managing Your Business

Wayne W. Eckerson
Author
Wednesday, March 22, 2006 2:00 PM

Wayne W. Eckerson , director of research for the Data Warehousing Institute, a leading association of business intelligence professionals, was online Wednesday at 2 p.m. ET to discuss his new book, "Performance Dashboards."

The book examines how businesses use technology to improve strategy and performance. He also explains the three different types of performance dashboards, and obstacles to overcome, such as company morale and privacy.

A transcript follows.

____________________

Wayne W. Eckerson: Hello,

Welcome to this chat. I'm honored that the Washington Post has asked me to answer questions from its readers.

_______________________

New Orleans, La.: Do we need a data warehouse in order to build an effective performance dashboard?

Wayne W. Eckerson: No, you don't need a data warehouse but if you have one, it makes things much easier. If the dashboard becomes the default BI tool or analytic application, you will more than likely need to build out a data warehouse or data mart to persist data for frequently asked queries

_______________________

Yorktown, Va.: In your opinion, to what depth should dashboard performance metrics be deployed (available/visible) in the corporation?

Wayne W. Eckerson: Hi Yorktown!

Great question. Ideally, a company should deploy dashboard metrics at all levels of an organization. Each level sees its own set of metrics that are aligned to the level above. In this way, metrics trickle down from the top and data rolls up from the bottom of the organization.

In terms of VIEWING metrics, however, it's best to let people only view metrics at their level of the organization. It's great if they can compare their performance to others at their level because this gets their competitive juices going, but it's often a cultural challenge to make change if the company has a culture where people hold on to data, and hoard it for political reasons.

_______________________

Upper Marlboro, Md.: Wayne, my apologies I haven't read your book ...yet ((smile)) ...so my question may be off base. This administration and the Office of Management and Budget are using a dashboard and traffic light type approach to evaluating performance for government programs. "Green" is good; and "Red" means the program, initiative etc are in trouble, likely to fail, etc. First, is this the type of dashboard you reference in your book? and If so, are dashboards the best approach to evaluating and managing complex programs, projects, etc.

Wayne W. Eckerson: Hi Upper Marlboro,

Good question. When I did research for the book, I saw so many different types of dashboards that I decided to cast my net fairly wide. The book describes operational, tactical, and strategic dashboards, the latter being the equivalent of a Balanced Scorecard, which is used to monitor the progress a company, unit, dept or individual is making towards achieving strategic objectives. The operational dashboard, on the other hand, is more like an automobile dashboard that displays real time information about an operation or process.

In general, any type of dashboard has a visual display mechanism like you describe. These vary greatly, some look more like dashboards, others like scorecards, and other like portals.

_______________________

Alexandria, Va. : I work for a trade association that represents owners and entrepreurs of small business. Overall, the owners often run blind, not knowing how they stack up against their competitors. We do the typical industry financial ratios, but I know they could use more measurement tools. How can a small business tap into performance dashboards without going broke? Are these tools just for the big players? If tools for small business are out there, how can I get the word to the masses?

Wayne W. Eckerson: Hi Alexandria, VA,

Yes, there are a lot of start-up companies offering easy to use, low-cost dashboard tools. Some of these tools have more glitz than substance. They demo great but you have to be a tad careful. They may not provide enough data to take action on the alerts that show up on the dashboard. They may also gloss over data quality problems that usually arise when trying to integrate data from multiple systems.

But the good news is that companies can get started with these tools without spending an enormous amount of money. (Of course, enormous is relative. The performance dashboards that I profile in my book cost into the millions, when you count all the infrastructure, but these new tools can be purchased and implemented for under $100,000 in general.)

_______________________

Philadelphia, Pa.: I look forward to reading your book. Would you please briefly explain what the performance measures are that you see as the ones that should best be considered in evaluating a company?

Wayne W. Eckerson: Hi Philadelphia,

My book has a list of evaluation criteria in the appendix. I'm currently turning this into an Excel spreadsheet that companies can use and modify to evaluate multiple products. We'll publish this on our Web site shortly. (www.tdwi.org.) Ping me if you want me to notify you when it comes out. (weckerson@tdwi.org.)

_______________________

Hyannis, Mass.: We're a nonprofit that is working on dashboard reports, currently we produce them by business unit. It would be ideal to have dashboards combined into a single report that is produced automatically through systems. Realizing this is an over-generalization, but how much should organizations budget (both cash and in staff time) to develop a dashboard reporting system?

Wayne W. Eckerson: Hi Hyannis MA!

The pricing for performance dashboards ranges the spectrum. You can implement dashboards manually using Excel and PowerPoint for virtually nothing except the time it takes people to update them (which can be significant!)

Many start-up vendors now offer dashboard tools for under $100,000, some for under $30,000, when fully implemented. But beware, you get what you pay for. If you are simply trying to make your spreadsheets prettier and more automated, it won't cost much at all. But you may not be delivering all the information people need to make decisions.

Ultimately, if you want the dashboard or scorecard to become the primary means by which you disseminate both historical and right-time data to all levels of workers to make better decisions and plans and work more proactively, then it's likely you will need to implement a data warehousing and business intelligence infrastructure that can help ensure the delivery of accurate and integrated that is both detailed and aggregated, you will need to spend a lot more money, closer to $500k to $1 million+. But then you'll get a real data management infrastructure that you can use to get a competitive advantage.

_______________________

Vancouver, Canada: Small organizations tend to rely heavily on MS Office programs. Do you think Excel is an adequate tool for creating dashboards?

In our office people love to use Excel, probably because they are comfortable with it and they feel productive using it. We have a BI system in place for data management and reporting, yet it is still common to see people working with report data in spreadsheets they have spent time creating themselves. What features can a BI system offer to get people to use it instead of Excel?

Wayne W. Eckerson: Hi Vancouver,

Sounds like you have a spreadmart problem. Spreadsheets being used as data marts. The problem with spreadmarts is that the data in them rarely synchs up - everyone uses different rules and methods to extract, transform, and display data. This means there is no single consistent view of information in the organization, which can be a real problem.

However, this is not to say that Excel is bad. Excel is great for what it is supposed to be used for: a modeling tool for creating planning scenarios. Many BI tools now integrate seamlessly with Excel so that you can manage data centrally and ensure its consistency and yet display the results in Excel and allow users to perform their analysis there.

_______________________

Fort Myers, Fla.: What tools do you see used most often to develop these dashboards? We've considered using HTML pages in Outlook Today for personalized dashboards.

Wayne W. Eckerson: Hi Fort Myers,

When I did research for the book, I looked for companies with mature dashboard implementations. At that point (2004), companies with 2-3 year old implementations had largely written the applications themselves. There really weren't many commercial tools available that were mature enough to warrant using.

Today, things have changed. Almost every major BI vendor (Cognos, Business Objects, MicroStrategy, ProClarity, Information Builders, Hyperion, Panorama) and ERP vendor (Oracle, SAP, Microsoft) and a slew of start-ups (Appfusion, Celequest, Theoris, QlikTech, Noetix, Corporate Radar, Corda) are delivering dashboard and scorecard tools.

There are submarkets for visualization tools (Tableau, Visual Mining) and scorecard tools (InsightFormation, Microsoft) and BPM tools (Hyperion, Cognos, etc.) I'm currently doing a market overview of all these products. It's like boiling the ocean of all analytical tools! Everyone has put a monitoring layer on top of their analytical, reporting, or managemente tools.

_______________________

Anonymous: What steps should businesses take to see that they not only recognize dashboard warnings but they then respond to the information and act accordingly? Too often I see companies that see the warning signs but ignore them.

Wayne W. Eckerson: Hi Anonymous,

Good question. I suspect two things are happening: 1) the alerts are not set finely enough so that people are either getting bombarded with too many or the alerts they see aren't really urgent 2) People aren't being held accountable for the metrics and the processes they measure.

A common mistake is that people publish too many metrics on a dashboard. They don't really prioritize metrics by aligning them with strategy and thus oversaturate users.

_______________________

Washinton, D.C.: Dashboard is neat and all, but I see lots of instances where workgroups are not willing to get involved in a project, until "everything is in tip-top shape" by others, for the fear that it will affect their performance matrices. Managers are pushed to come up with man-hour vs dollars saved/made that mirrors the big debate on body-count vs success in war-on-terror. Directors are acting as if throwing twice more people needs to reflect 2 times efficiency in KPI. It all ends up with the 30% real workers having to do overtime, which is free, to get the big bonus for the big shots. How much longer will the buzz about the dashboard last?

Wayne W. Eckerson: Hi Washington DC,

There is hype out there on dashboards... mostly what I call quickee dashboards.

But defining the right metrics and rolling them is a real art. First, I wouldn't recommend attaching incentives to metrics until they are fully vetted, everyone buys into them and knows how to affect them. Otherwise, you court chaos!

Second, you can never get the perfect metric the first time. You need to get 80% of the way there and implement it to see what action it drives or doesn't drive, and tweak it correspondingly.

It's very difficult to define accurate metrics to measure business processes. There are so many nuances to business processes and it's very easy to game the system. We are getting into the realm of performance measurement here.

Also, metrics have a lifespan. When you first roll them out, they'll have a huge impact (hopefully positive!) and after time, you'll hit the law of diminishing returns where the expenditure of effort doesn't produce equivalent or requisite gains in performance.

_______________________

Apple Valley, Calif: 1. Do you think the growing popularity of dashboards (seems to me the market languished for several years) has been driven mainly by Sarbox issues?

2. Dashboards are often faulted for being rear-view mirrors on performance, but vendors seem to be reacting to that with new offerings. Any comments?

3. One of the downsides to dashboards is the time required to set them up and show the execs how to use them. How do you recommend IT handle this, and about how much total IT staff time is involved per dashboard? 4-5 hours? (interviewing and configuration)

Wayne W. Eckerson: Hi Apple Valley!

1. Dashboards are just gaining momentum, if you ask me. People finally realize this metaphor is the way the majority of users in an organization want to consume information. Monitor, then analyze, then view detailed data or operational reports. Basically, dashboards are exception reports... but done right with right-time information when required and displayed graphically using symbols and color-coding.

2. Scorecards tends to deliver historical information but dashboards are more operational in nature. More and more people want to view information in a more timely manner so they can work proactively. You can deliver right-time information via your data management infrastructure or using BI tools equipped with EII capabilities. (EII equals the ability to query multiple sources and pulling the information together on the fly -- powerful yet it has some scalability limitations.) Many new dashboards are built on EII technology.

3. Demoing a dashboard is easy. You can mock up something in any BI tool or visualization tool, or even Excel. But gathering requirements and sourcing data is the hard part. Some executives want to rush into dashboards without really knowing what they want to measure. Or they know their measures (hopefully emanating from a strategic planning session) but there is no data (or good quality data) to populate the metrics. One vendor says 10 metrics in three days is a good rule of thumb, which includes sourcing the data and defining the calculations, but not the requirements. In a data warehouse environment, when you add a new source of data or subject area, it usually takes 5 people three months from start to finish --gathering requirements, sourcing data, cleaning, transforming and loading data, creating reports, training users, etc.

_______________________

Apple Valley, Calif.: Are there any decent open source dashboards?

Wayne W. Eckerson: I haven't come across any yet.

_______________________

Wayne W. Eckerson: Thank you for your questions! Feel free to email me with additional questions at weckerson@tdwi.org.

_______________________

Editor's Note: washingtonpost.com moderators retain editorial control over Live Online discussions and choose the most relevant questions for guests and hosts; guests and hosts can decline to answer questions. washingtonpost.com is not responsible for any content posted by third parties.