The big idea: Projects are often late, over-budget or under-delivered. Can companies set more realistic project goals?

The scenario: The New York City Department of Parks and Recreation is responsible for maintaining more than 1,700 city parks and recreational facilities on more than 29,098 acres. The department completes about 150 capital construction projects a year. In the fiscal year 2011, its spending on capital projects was $496 million. A 2013 audit found that 47 percent of the projects were not completed within their originally scheduled time frames and that 10 percent exceeded their original budgets, with increased project costs of $13 million.

Despite awareness and project-management techniques, many organizations grapple with projects delivered late, over budget or with reduced scope. Daniel Kahneman from Princeton University, with his longtime collaborator Amos Tversky from Stanford, called this phenomenon “the planning fallacy.”

Cognitive biases such as optimism bias (the tendency of individuals to expect positive outcomes from their actions) and overconfidence have been suggested as causes of the planning fallacy. There is a growing body of evidence, collected by researcher Bent Flyvbjerg at Oxford University, that optimism bias is one of the most important biases when it comes to factors that affect the quality of forecasts in project planning.

Other explanations of the fallacy highlight more intentional and deliberate considerations on behalf of the planner — such as incentives, organizational pressures and strategic deception.

Kahneman and Tversky, and later Dan Lovallo, have suggested that taking an outside view to forecasting helps reduce the planning fallacy. Many project-planning frameworks used by firms today, such as the critical path method, take an inside view to planning, whereby project goals are set by focusing on the specifics of the project at hand. Taking an outside view suggests that when forecasting the cost or duration of a project, it is beneficial to consider the outcomes of previous projects. Moreover, historic forecasting errors and their distribution can be valuable when trying to calibrate a future forecast if a set of similar projects can be found.

The resolution: An analysis of historical data from 1,800 New York Parks and Recreation projects suggests that the size of a project, the complexity of work required and type of subcontractors involved can characterize a set of similar projects — a reference class. Details drawn from this class can inform the goals we set for the new project before planning begins.

The lesson: Organizations that suffer from the planning fallacy have benefited from taking an outside view of project planning to produce better estimates. By analyzing historical data and statistics from previous, similar projects, project managers can set better goals for the future.

— Yael Grushka-Cockayne

Grushka-Cockayne is an assistant business professor at the University of Virginia Darden School of Business.