A Cabinet-level committee unanimously recommended yesterday that President Reagan let Japan decide whether to continue the so-called voluntary restraints on Japanese auto sales to the United States for a fifth year, administration sources said.

The recommendation, the first administration decision in one of the most explosive trade issues of recent years, will become an element in the overall review of the United States' trade relations with Japan that resulted from President Reagan's Jan. 2 meeting with Prime Minister Yasuhiro Nakasone.

Reagan "is more concerned about opening Japanese markets than in how to advise them on auto restraints," a White House official said. "He is looking at this in the overall trade context. He is comfortable with not moving specifically" on auto restraints.

Thus the recommendation by the Cabinet Council on Commerce and Trade will not go directly to Reagan, but instead will be tied to the administration's larger efforts to open Japanese markets to highly competitive American products. That program is being coordinated by Secretary of State George P. Shultz and National Security Adviser Robert C. McFarlane.

The recommendation that the president take an essentially passive stance on import quotas for Japanese cars came as key congressmen, organized labor and three of the four major U.S. auto makers pressed the White House to continue past March 31 the restraints that limit Japan U.S. auto sales to 1.85 million cars a year.

While presidential spokesman Larry Speakes refused to reveal yesterday morning's decision of the Cabinet council, he hinted broadly that Reagan may do nothing when the restraints end.

"It could be no decision" is required, he said. Extending the quotas for a fifth year "is a decision for the Japanese to make," he added, repeating the standard White House position that the quotas, which grew out of intense negotiations between the two governments in 1981 and 1983, were installed voluntarily by Tokyo.

Other sources, revealing the Cabinet council's recommendation, said one factor was the healthy state of the U.S. auto industry. The Big Three American auto makers -- General Motors, Ford and Chrysler -- reported record $9.8 billion profits for 1984.

Pressures on the White House to push to renew the auto quotas intensified with the publication last month of the United States' record $123.3 billion trade deficit, with Japan responsible for the largest single share of it, $36.8 billion.

Japan's auto sales of $20 billion amounted to more than half of the trade deficit and one-third of all Japanese exports to the United States. But the International Trade Commission estimated the total deficit would have been $4 billion higher without the restraints.

The cost of auto restraints, however, has been high. The ITC reported last week that the four years of quotas cost the American consumer $15.7 billion while saving 44,000 auto workers' jobs. The restraints also raised the cost of each Japanese car sold in the United States by about $1,300 and allowed American auto makers to charge $600 more for their products last year, the ITC found.

"Reagan knows there are two sides to this in the United States," a White House official said. It doesn't really matter what Reagan said because the Japanese know they have to use restraint, he said. "If they don't and if they flood the U.S.," Congress would respond with protectionist legislation, he said.

Japanese auto makers and government officials, in conversations with the Reagan administration, have said they want the limits lifted, and have indicated they will police themselves by keeping their exports to reasonable levels.

"I'm not sure I want MITI Japan's Ministry of International Trade and Industry to decide our fate," commented one Ford executive.

Congress already has two bills before it setting mandatory quotas on Japanese auto sales that are more restrictive than those presently in place.

Sen. John C. Danforth (R-Mo.), moreover, plans to introduce today a sense-of-the-Senate resolution, with at least 30 cosponsors, calling on the president to leave the current auto restraints in place until Japan significantly increases its purchases of American products. A similar resolution will be introduced in the House by Minority Leader Robert H. Michel (R-Ill.)

American auto makers are split on whether to continue the restraints, with General Motors -- which has made arrangements to import small autos from Japan -- the only one to favor ending them.

Ford, Chrysler and American Motors, as well as the United Auto Workers, want the quotas continued. Those three automobile companies have threatened to drop planned investment in American facilities and to begin importing larger numbers of foreign-made cars if the restraints are not retained.

"We at Chrysler would have to rethink our long-range investment plans, which currently call for nearly $10 billion to be invested in the United States over the next five years," Chrysler Chairman Lee A. Iacocca said in response to reports that the Cabinet council had not recommended that quotas be continued.