The Justice Department has created a new investigative unit of 15 prosecutors and investigators to ferret out fraud in the Defense Department.

The investigators, who are to move soon to their own office in Alexandria, come from Justice and each of the three military services. They are to pursue leads from Defense Department auditors and officials, defense contractors, and callers to a military fraud hot line.

Officials said they have long believed that military procurement is a major source of fraud, if only because of the large amounts of money involved. Spending for weapons and military supplies during the current year is expected to reach $80 billion, or more than $200 million a day.

The General Accounting Office, the auditing arm of Congress, has issued numerous reports on waste in the military but has not totaled it. The FBI conducted 1,071 investigations into defense operations in fiscal 1981, resulting in 72 convictions, assessment of $348,800 in criminal fines and recovery of $7.6 million in civil penalties.

But because of the complex nature of such cases and jurisdictional rivalries, the Justice Department has not been effective in prosecuting them, according to Robert W. Ogren, chief of the department's fraud section.

"There have traditionally been real problems with one service working with another," he said. "The investigations were encumbered by the technical nature of the cases and the bureaucratic labyrinth in dealing with the Pentagon."

"In any business," he added, "a fraud or loss level of 1 to 3 percent is not regarded as wild. When you talk Defense procurement, 1 or 3 percent is hundreds of millions of dollars."

Ogren, a former prosecutor in the U.S. Attorney's office in Washington, chose Richard A. Sauber, 33, a former Rhodes scholar, to head the new investigative unit.

Although Sauber said he expects that his staff will develop expertise in military procurement rules and methods, much of the work is to be done by Defense Department investigators or auditors and FBI agents.

In the past, Sauber said, defense contractors caught cheating the government would offer to pay half of the alleged overcharge, and the cases were dropped. "The prosecutors would say, 'You got the money back, why worry about it?' It's not a violent crime you're looking at. You're looking at a six- to 12-month investigation and a 30-day trial."

Others expressed skepticism about the impact of the new unit.

Calling it "eyewash," A. Ernest Fitzgerald, the Air Force cost-cutting analyst, said, "The Justice Department is part of the problem . . . . Justice just hasn't gone after waste and abuse in the Pentagon. They represent the other side," referring to the department's role of defending government officials who are sued.

"They've been simply unwilling to move on allegations concerning major Defense contractors," said Richard F. Kaufman, an aide to Sen. William Proxmire (D-Wis.).

But retired Navy admiral Hyman G. Rickover said, "It's about time that the government, which is spending all this money, is going to find out where it's going."

Thomas L. Patten of the Washington office of Latham Watkins & Hills, who represents defense contractors, said, "I'm not saying there aren't things out there worth chasing down, but I fear we're killing a fly with a sledgehammer . . . . There is no reason to believe that defense contractors are any less honest or deserving of trust than any other citizen."

The new unit, established last October, has brought two indictments, one for forging and inflating prices on purchase orders for Army commissaries in Europe and another resulting from an alleged kickback scheme.