The Labor Department "lacks credibility" because of its failure to police the nation's $900 billion pension and health insurance system effectively, and because of its delay in issuing more health and safety rules to protect workers, according to an unpublished study by the General Accounting Office.

The GAO "management review," a copy of which was obtained by The Washington Post, also concludes that the department's Mine Safety and Health Administration is severely understaffed and therefore is failing to perform nearly 40 percent of mine inspections required by law.

Labor Secretary William E. Brock received the 112-page GAO "briefing" paper last month. It outlined what the GAO called serious problems that stem in part from the department's efforts to accomplish too many goals with a shrunken staff and with little long-range planning.

"They didn't give the department very high marks," acknowledged David Demarest, deputy undersecretary for public and intergovernmental affairs. Demarest said the GAO study was made long before Brock took office in May, and that Brock has already formed several task forces to implement changes that GAO will recommend in a forthcoming draft report on the same issues.

Among other findings of the preliminary GAO study:

*The $15 billion unemployment compensation system remains "highly vulnerable" to fraud and inefficiency because the department does not adequately monitor or audit the state-administered system.

*The $2 billion Job Training Partnership Act program, which replaced the scandal-ridden Comprehensive Employment and Training Act (CETA), is "vulnerable" to the same fraud and abuse that scuttled the $9 billion CETA program unless the department tightens its monitoring.

*Reductions-in-force have hurt labor programs, by cutting manpower and by forcing the shift of workers into unfamiliar new jobs. Nine rifs since 1982 have resulted in 1,442 separations, 1,149 demotions and 1,419 reassignments at the department.

*Recruitment of qualified employes is a "special problem" for the Occupational Safety and Health Administration, the MSHA, the Bureau of Labor Statistics and other divisions because government salaries lag $5,000 to $7,000 behind private-sector salaries for scientists, engineers, economists and similar professionals.

*The Labor Department does not always plan its computer and information systems carefully. For example, the GAO said, the department spent $90,000 on computer terminals for its personnel system and later found they were incompatible with the computer system it had in place.

The GAO sent questionnaires to 183 department managers (98 percent responded) and interviewed nine former department officials. The GAO also contacted 26 groups that deal with the department, ranging from the AFL-CIO to the U.S. Chamber of Commerce.

The GAO concluded that "the most significant challenge the department faces is protecting the worker without overburdening business." The GAO said that organized labor believes the department is failing in its efforts to protect workers, while "conversely, business groups expressed satisfaction with the department's current philosophy of less federal intervention."

As part of its shrinking role, the Labor Department has cut its staff from 23,900 in 1980 to 18,600 in 1985, with a cut in its "controllable costs" of 36 percent, from $11.9 billion to about $7.6 billion, the GAO said.

For instance, the Office of Pension and Welfare Benefit Programs has only 200 investigators who are responsible for policing 915,000 pension and 4.5 million health and welfare plans. "After 11 years, the enforcement program lacks credibility . . . less than 1 percent of the plans are reviewed each year," the study said.

Brock plans to beef up pension oversight, department sources said, by elevating the office directorship to the status of an assistant secretary and by filling the long-vacant post with Dennis M. Kass, 34, a former White House aide who managed a New York investment firm. David M. Walker, acting director of the Pension Benefit Guaranty Corp., is expected to be named Kass' deputy.