When Joseph H. Sherick became inspector general of the Defense Department last year, he had no trouble picking the target of his first major investigation.

"I knew that we didn't have an adequate audit follow-up system in place," Sherick explained. "So I told my staff to collect all the recent internal audits that had been done on the Pentagon. And then I said, 'Now, we're gonna go out and see if we can find the same thing, but we're gonna stop it because we're gonna make sure that corrective action is taken.' "

What Sherick discovered almost immediately was that nearly 140 audits had identified problems in the pricing of spare parts. What followed was an agency-wide investigation directed by Sherick that exposed dozens of "horror stories" in which DOD paid exorbitant prices for usually inexpensive items, such as $1,100 for a plastic cap for a stool worth less than $1.

At DOD, which spends $600 million a day and conducts business with more than 100,000 firms each year, the problem is not locating leaks in the dike, but deciding which holes must be plugged first.

Sherick, 60, an attorney and career civil servant, said he still is working off his original list of old audits that have been largely ignored.

He has launched agency-wide audits of claims and costs of the Federal Employes' Compensation Act, cash management, inventory control, quality of service at military hospitals and workmanship by DOD production contractors.

"We are the biggest pot in town . . . if people are gonna steal from us, it's gonna be big bucks, not thousands, but millions," he said. "So I try to find where we are the most vulnerable and then I look to see if it's an across-the-board problem or something that a specific service, like the Army, can deal with. If it is agency-wide, I go after it."

Congress took four years to decide how much power to give the Pentagon's inspector general. The House wanted him to have complete autonomy, but the Senate said an IG might accidentally leak national security secrets if he could snoop anywhere.

Under their compromise, the secretary of defense has the authority to block probes in five areas that might involve intelligence-gathering operations. But it requires the IG and the secretary to tell Congress when an investigation has been blocked.

Sherick said Secretary Caspar W. Weinberger has not complained about any inquiries. "His only comment to me has been to 'Go get 'em.' "

"If there are going to be questions of jurisdiction, I want them to come up right now," Sherick said, "because I've been around this agency a long time, and I want to set the precedents now while we have a president and secretary who support what we are doing."

Sherick described himself as a "kid from the back streets of Philadelphia." He joined the Navy at 17, returned to Philadelphia after World War II and earned business and law degrees from Temple University while working as a supply and budget officer at the Army arsenal there.

After graduation, Sherick moved to the Pentagon as an Army budget specialist, then went to what is now the Office of Management and Budget. He was chosen as the first civilian comptroller of the Defense Nuclear Agency in 1966, but returned to the Pentagon in 1968.

His staff is the largest of the 17 inspector general staffs in the government, nearly 1,000 persons. He also has access, if Weinberger approves, to the nearly 19,000 auditors and inspectors who work for the service branches or for the Defense Contract Audit Agency.

Between October 1983 and March 1984, Sherick said, Pentagon auditors, mostly at the DCAA, helped the government avoid more than $3.5 billion in costs and collected $10.8 million in fines and penalties.

"For every dollar wasted at DOD, I believe only 2 cents are stolen; the rest is lost because of mismanagement," said Sherick, who has concentrated on correcting mangement problems.

Sherick keeps a can of dog food and a huge dog collar in his office: gifts that arrived after President Reagan said he wanted the IGs to be "as mean as junkyard dogs" in uncovering waste, fraud and abuse.

"Our job is to go out and embarrass people," Sherick said. "My motto is that you have to be competent, courageous, because on a lot of these things you have to stand up and fight with everybody and you have to have compassion . . . ."

But some Pentagon critics contend that Sherick is "too nice."

"Everyone always describes Sherick as a 'nice man,' " said Dina Rasor, director of the Project on Military Procurement. "Those are not words that they use to describe Ernie Fitzgerald," one of the Pentagon's best-known whistle blowers. "The IG shouldn't be a job for a nice guy."

Sherick said his biggest job is to make the bureaucracy realize that he is not a passing fad.

"I am trying to establish an institution here. The IG is going to be here for a long, long time and he is someone you are going to have to deal with. That is something the bureaucracy has to accept, but you accomplish that by doing good work and having a good track record."

If Sherick can prove that the IG's office is as "mean as a junkyard dog," he said there will be fewer holes in the Pentagon dike to fill.

"People must realize that someone is watching them."