A small unit in the Labor Department with fewer than 500 employees has challenged many larger agencies for the dubious distinction of producing the greatest volume of complaints from the public to congressional offices about slow and shoddy service.

The unit is the Office of Workmen's Compensation Programs and it is among the most frequently criticized agencies of government in a survey of congressional offices being taken by President Carter's reorganization task force.

Members of the House and Senate and their staffs were asked by Richard Pettigrew, assistant to the President for reorganization, to list the federal programs and agencies "which currently generate the most dissatisfaction, frustration and confusion on the part of your constituents."

With the deadline today, replies have been received from about 50 out of 535 members of Congress. As might be expected, big agencies with widespread public contacts, like the Postal Service, and such controversial bodies as the Occupational Safety and Health Administration drew a great many gripes.

But right up there near the top of this reverse hit parade of the bureaucracies is the Office of Workmen's Compensation Programs, which handles the job-injury problems of federal employees and some 350,000 others.

Complaints against OWCP turned up on almost half the returned questionnaires. They were perhaps best summarized by the comment from one House member who said the agency was noted for "extreme delays in the processing of claims; rude treatment to claimants; incompetent employees."

A Labor Department spokesman said that the problems of OWCP had been the subject of a six-month internal investigation by a task force which made more 100 recommendations earlier this year.

Last week, assistant Secretary of Labor Donald Elisburg told the American Bar Association convention that the Carter administration was "putting a lot of new people and funds" into OWCP to help it catch up with what he called "an extraordinary increase in its workload."

OWCP's main job is providing monetary compensation, medical assistance, vocational rehabilitation, funeral expenses and survivors' benefits for injured federal employees.

It also runs the "black lung" program for miners, which was also the subject of numerous complaints in the Pettigrew survey.

One senator said "a constituent called earlier this week to say that her parents have been waiting for more than four years for a decision their claim" for black lung compensation.

Compensation for disability appears to be a troublesome area no matter which government agency is involved. While the Social Security Administration was complimented for its handling of some programs, its disability insurance decisions were also the subject of many gripes.

Other agencies and programs that appeared with frequency on the list of complaints were the Immigration and Naturalization Service, the Internal Revenue Service, and the Environmental Protection Agency.

Many housing program came in for criticism, and one member of Congress commented acidly that the Department of Housing and Urban Development "never answers calls."

Pettigrew also asked the congressional offices to list the agencies "which your constituents find most responsive." Many members replied that frankly, they didn't hear about those; they only heard the horro stories.

But at least one big agency with millions of public contacts each year was singled out for frequent praise - the Veterans Administration.

One House member wrote of the VA: "The regulations are plain, the forms are clear, service is available and reasonably prompt, counselors are helpful and well-trained."

Other agencies that drew more than scattered mention as competent and helpful were the Passport Office of the State Department and the Agriculture Department's Forest Service.

The military services were praised for the efficiency of their congressional liaison offices in handling constituent problems.

But the overall impression of the survey has rather negative. "It would be helpful," said one congressman, capsulizing many similar comments "in almost all federal agencies, if employees regarded themselves as public servants rather than being resentful of the imposition of their time and energies required to answer both telephone and written inquiries."

Pettigrew said the survey results would be helpful to Carter's reorganization team in identifying "trouble spots" which should be given priority in the President's effort to remodel the executive branch for greater economy and efficiency.