While the Reagan administration has become increasingly concerned about the threat of terrorist attacks on government buildings, the General Services Administration is taking steps critics say will weaken the Federal Protective Service that guards most of them.

The GSA wants to fire up to 350 of the service's mid-level supervisors this year and transfer the day-to-day supervision of most of its officers to GSA building managers. The service's 2,200 employes, along with 4,411 private guards, provide security at 7,200 government buildings.

The cuts were developed by top agency officials in February as a way of saving $10.1 million this year. In a memo to agency officials in March, then GSA Administrator Ray Kline wrote that the agency must "move quickly to eliminate any supervisory layers which do not add distinct and measurable value."

"It is obvious that we think that there are some efficiencies to be gained," said Charles S. (Terry) Davis, associate administrator for operations, and a co-author of the proposal. "Else, we wouldn't be doing this."

But Rep. Barney Frank (D-Mass.), chairman of the House Government Operations subcommittee on manpower and housing, said he is ready to call for congressional hearings to explore the merits of the cuts.

"It is a mistake," Frank said. "I think that GSA ought to be upgrading the officers some to make sure that any threats can be repulsed. It's a very bad idea."

Loretta Ucelli, a spokesman for the American Federation of Government Employees, said the plan "underscores our increasing concern about the way the government is managed and its misplaced priorities."

Ucelli said the Reagan administration has been determined to make the White House a fortress, but "they want to lessen the security where the federal employes and the public are concerned." At the request of the administration, the GSA this year installed concrete barriers at the White House, the State Department and several other buildings that house security-conscious agencies.

Although Dwight Ink, GSA's new acting administrator, has said he intends to follow through on the recommendation, the new public buildings commissioner, William F. Sullivan, who oversees the Federal Protective Service, has asked for another month to study the proposal before it is implemented.

"I need to be certain of the impact that the cuts might have on morale and effectiveness or on our responsibility to protect federal employes and buildings," Sullivan said.

"We must look more into the question of how we are applying our resources," he added. "Should we try to cover the entire nation with a Federal Protective Service? We've got to restudy the reporting relationships; should they report to building managers? Is that a wise course of action?"

John B. Osborn, deputy assistant commissioner for the protective service, said he is advising Sullivan that there isn't as much fat in the supervisory layers as Davis has suggested. "You can always go out and cut back on any operation," Osborn said. "But that doesn't mean that there's fat. In this case, from my view, this is just a budget cut."

Isabelle Cummins, a legislative aide to Frank, said, "If most of the employes in the unit were white, they would be more careful. We have a situation . . . where the administration is looking to affect those who have the lowest voice."

Union spokesman Ucelli also contends that minorities' jobs are often the first ones targeted before the employe's opinion is sought.