The Reagan administration was warned yesterday that its proposal to ease antitrust rules for joint research ventures faces trouble on Capitol Hill if it is part of a larger package that includes ending triple damage penalties for most civil antitrust cases.

Administration witnesses told the Senate Judiciary Committee that both proposals are needed to improve American industry's competitiveness in the global market.

Sen. Howard Metzenbaum (D-Ohio) said the administration proposal to remove triple damages for all but the most anticompetitive offenses would "decimate the antitrust laws of the country."

Companies that violate antitrust law now can be penalized by three times the amount of damages their anticompetitive actions were found to have cost. The law provides for treble damages as a deterrent.

The administration proposal would grant joint R&D projects immunity from private and government antitrust suits and triple damage awards if the ventures meet certain requirements. If they turn anticompetitive, however, the government can stop the ventures through a federal court injunction. Outside the R&D area, it would eliminate triple damages in all but the most flagrant antitrust violations, such as the formation of cartels or price-fixing.

While not fully agreeing with Metzenbaum, Sen. Charles McC. Mathias warned both Commerce Secretary Malcolm Baldrige and Assistant Attorney General William F. Baxter that the administration's antitrust amendments designed to encourage joint R&D ventures will be jeopardized if they are tied to a plan to end triple damages.

"When you attack triple damages," warned Mathias, "you are attacking one of the historic bastions of antitrust. You are going to arouse the interest, the opposition and the passions of people inside the Senate.

"You need a minimum of 60 votes to pass the bill," he continued. "I'm not sure I can count 60 votes if you get into that kind of a scrap around here."

Retired Adm. Bobby R. Inman, head of a high tech research enterprise formed by leading American electronic companies, agreed with Mathias' assessment. Inman, former director of the supersecret National Security Agency, recommended against loading the bill "like a Christmas tree."

The administration has not yet submitted its antitrust package to Congress, though it has circulated drafts on the Hill and Baxter has described it in talks to business and legal groups. But this was the first time he and Baldrige defended the plan before a congressional committee.

The rationale for both the R&D changes and the rest of the package appeared to center on the need to increase America's competitiveness in the world economy.

Baxter made it clear he considers antitrust exemptions for joint R&D ventures "a minor problem," while the key to increasing American competitive strength lies with ending triple damages.

Baxter told the committees the chilling effect of triple damages for minor antitrust violations does "more damage by a wide measure" than its force in deterring the serious anticompetitive violations.

"It's misperceived to be a protection for the consumer," said Baxter.

In his testimony, Baldrige focused on how to ease the way for American companies to join in research projects such as Inman's Microelectronics and Computer Technology Corp. (MCC), formed by 13 U.S. high-tech firms. The Commerce secretary called joint R&D programs "procompetitive" because "they reduce duplication, promote the efficient use of scarce technical personnel and help to achieve desirable economies of scale."

He said major trading partners and overseas competitors such as Japan, France and the European Economic Community allow joint R&D under their antitrust laws. American companies, however, hesitate to get involved because of the risk that joint ventures will be found to violate antitrust laws and the companies could face triple damages penalties and criminal sanctions.