As we’ve discussed before, the number of Americans in the labor force — that is, the people who either have jobs or are actively looking for work — has been dwindling in recent years. Some of that’s been due to ordinary demographics: America’s getting older and more people are retiring. Some of it’s been due to the grim economy, which has dissuaded many people from even bothering to look for jobs.

And now we can see this in glorious chart form, thanks to Dave Altig of the Federal Reserve Bank of Atlanta. Since the recession began, the “labor force participation rate” has fallen from 66 percent to 63.6 percent. Altig estimates that about 0.9 percentage points of the drop was due to demographic factors, while about 1.5 points was due to the horrible economy. In other words, if the recession had never happened, the labor force participation rate would probably be about 65.1 percent:

So that’s about 1.5 percent of the eligible population — or about 3.6 million workers* — who have been elbowed out of the workforce by a subpar economy that didn’t have enough jobs for everyone. (Although note that these subpar conditions began back in 2003 and accelerated with the recession.) So how many of these people will actually return to the workforce if the economy ever picks up steam again?

That’s a tricky question because, as economist Brad Delong points out, many of those discouraged workers are in danger of losing their skills and work contacts and becoming “part of the ‘structurally non-employed’ who we will never see back at work, barring a high-pressure economy of a kind we see at most once a generation.”

Here’s another graph, from Julie Hotchkiss of the Atlanta Fed, looking at what labor-force dropouts between 25 and 54 are actually doing once they leave:

There’s been a big surge in workers going back to school — it’s likely that they’ll be back in the work force someday. But there’s also been a big surge in workers going on disability, as eligibility has gotten easier. Workers on disability are unlikely to come back. There’s also the mysterious “other” category — it’s not clear whether these workers will ever return to the workforce.

The question of how many labor force dropouts come back to the workforce isn’t an idle one. It will affect how many jobs we need going forward. After all, if the labor force swells, then the U.S. economy will have to add more jobs to keep up.

Altig estimates that if the labor force participation rate stays where it is currently, then the U.S. economy will need to add just 144,000 jobs per month to get down to 7.5 percent unemployment by the end of 2013. On the other hand, if the labor force grows back to 65.1 percent — say, because a vast reserve of heartened workers start scanning wants ads again — then we’ll need to add 304,260 jobs per month to get to the same level.

* Correction: It’s 3.6 million workers missing from the labor force, not 4.5 million as originally stated.