Nonprofit and volunteer organizations cannot fill the gap created by federal cutbacks in social programs over the next three years because they will lose $33 billion in federal funds themselves, the Urban Institute said yesterday.

In what it called the first detailed study of the country's nonprofit groups, the Washington-based independent research organization said only an unprecedented -- and unlikely -- jump in private giving will allow such groups to keep going at current levels.

Even excluding churches and synagogues, nonprofit groups that provide services employ 4.4 million people, five times as many as the auto industry, yet have never been counted as major financial actors, the study said.

Neighborhood clubs and day-care centers, museums and medical complexes all get more money from the federal government than from private givers: $40.4 billion in fiscal 1980 versus $25.5 billion from foundations, corporations and individuals, the study continued.

The Reagan administration has said it expects nonprofit and volunteer groups to help pick up programs dropped from the federal budget. But this view "is not correct because it overlooks the relationship between the nonprofit groups and public funding," said Lester M. Salamon, who co-authored the report with Alan J. Abramson.

Just to provide current services in the face of federal cuts, nonprofit groups would need 24 percent more from private sources this year than last year, and 40 percent more in 1983. To begin filling the $115 billion gap that Reagan budgets will leave by 1985 in areas where nonprofit groups are active, the increase would have to be 60 percent this year and 147 percent in 1985, the study said. But the biggest increase in philanthropy ever recorded was a 12.4 percent rise in 1981.

"Nonprofit organizations by 1985 will be asked to take up a much heavier burden but to do so with much lower revenues," Salamon told a news conference. "Despite its avowed intention to increase the role of private institutions," the study said, "the Reagan economic program paradoxically threatens to place these organizations further in the hole and widen the service gap."

Salamon admitted that there is room for streamlining and increased efficiency in many nonprofit groups, but said, "The question is one of degree." He added that the President's Task Force on Private Sector Initiatives, launched last December, had made "real efforts" to promote corporate giving, but pointed out that corporations provided only $3 billion of the $25.5 billion donated in 1980.

Hardest hit among the nonprofit groups would be social-service providers, if the proposed cuts materialize. They make up 41 percent of all the 103,000 charitable service organizations that existed in 1977, the most recent count, the study said. (Another 62,500 nonprofit groups are labor unions, political, business and professional associations, sports clubs and other organizations not covered in the study.)

Social-service providers would shoulder $14 billion in cuts by 1985. Community development and education/research groups each would lose $5 billion, while health care groups would drop $8.1 billion and cultural and international aid associations would also be cut. Together their proposed federal funding would be 27 percent less in 1985 than in 1980, the study said.

Salamon said the picture showed "a clear policy . . . affecting social welfare" programs, even though this is the area where need is likely to be greatest because of the recession.

Although Congress could prevent all this, it went along with 75 percent of the proposed changes in the first 1983 budget resolution, the study said. State and local governments might opt to pick up some of the funding slack, but economic hard times make that unlikely, it said.