President Reagan decided yesterday against asking Japan to renew its quotas on automobile exports to the United States and said he hoped his action on behalf of free trade would spur Tokyo to drop its barriers to U.S. products.

The decision lifts four years of restraints on imports of Japanese cars and is sure to increase Japan's share of the American auto market from its present 18.6 percent to as much as 33 percent. But White House officials, asserting that American car makers are now strong enough to compete, said automobile prices should drop with the increased competition.

American auto makers "should be able to provide an economical car and compete with the Japanese," said White House Chief of Staff Donald T. Regan. If domestic manufacturers "give the American public what it wants in the way of automobiles" the Japanese won't boost their sales here, he added.

The so-called "voluntary" restraints were imposed under pressure from the United States in 1981, and in their fourth year, scheduled to end March 31, the Japanese had been restricted to imports of 1.85 million cars. This could jump by as many as 1.1 million cars by 1987, according to a Commerce Department analysis prepared for the president. Other estimates give lower figures, and private Japanese envoys suggested the additional imports would be no greater than 650,000.

The president faced tough choices in deciding whether to ask Tokyo to continue restraints and whether to link any White House action to Japanese market-opening moves agreed to in January by Reagan and Prime Minister Yasuhiro Nakasone -- especially in light of the United States' $37 billion trade deficit with Japan.

The president acted after receiving unanimous recommendations from his principal foreign policy and economic advisers, including Secretary of State George P. Shultz, Commerce Secretary Malcolm Baldrige and U.S. Trade Representative William E. Brock, that the quotas should end.

White House spokesman Marlin Fitzwater said the president based his long-awaited decision on the added costs to consumers from the import limits; the new prosperity of a "healthy and vibrant" American auto industry that had racked up record profits totaling $10 billion last year, and the administration's commitment to free trade.

"This is now doing what we've long thought should be the right way to do things -- leave it up to the free-market choice, leave it up to the consumer," said Regan.

Fitzwater denied that the restraints will be lifted in exchange for Japan's opening of its markets in four areas agreed to in the talks between Reagan and Nakasone -- telecommunications; wood and paper products; sophisticated electronics, and pharmaceuticals and medical equipment. Regan said he expects "good results" out of the ongoing trade talks before the two leaders meet again in May at the economic summit in Bonn.

"There are no hidden deals" nor "any hidden smiles or winks," said Fitzwater.

"We are simply setting a climate and a policy and hoping that it will have an effect on the way they the Japanese deal with us," he added.

The president raised the issue of a trade-off with the Japanese in his statement when he said, "In taking this action, I hope that we can look forward to reciprocal treatment by Japan concerning the high-level discussions under way between our countries in the weeks and months ahead."

The president's decision now throws the ball in Japan's court, which, according to reports from Tokyo, leaves the Japanese unsure about what free trade might bring. They had been hoping for some hint from the Reagan administration as to what was expected from them.

Reagan's statement drew a positive response within the Japanese government Saturday morning. "This is a kind of rollback action against protectionism," a Foreign Ministry official said.

But the statement apparently did not wipe away Japanese feelings that car exports must remain under some type of restraint to keep relations with the United States on an even keel.

"The Japanese government thinks that each individual Japanese auto manufacturer will be prudent enough in their exports, considering the present situation" in the United States, the official said.

There were conflicting reactions here concerning how Reagan's announcement might affect ongoing talks toward facilitating sales of U.S. telecommunications equipment in Japan.

Some Japanese policy makers were said to feel it will strengthen the U.S. stance. Others, however, contend that the auto restraints are an artificial and unfair control and that U.S. suggestions that they be lifted require no concession in response from Japan. The effects of four years of auto restraints have been mixed. The International Trade Commission estimated last month that the import restraints cost American consumers $15.7 billion over the four years, with the costs of Japanese cars $1,300 higher last year than they would have been without quotas. The quotas, however, kept an estimated 1 million Japanese cars out of the country, saved 44,000 auto makers' jobs and lowered the already booming U.S. trade deficit with Japan by $4 billion, the ITC said.

President Owen Bieber of the United Automobile Workers, however, said ending the restraints will cost 200,000 jobs and set back the auto industry recovery. Along with the job loss, Sen. Donald Riegle (D-Mich.) said the tax loss from the lack of restraints could add $30 billion to the federal budget deficit and $10 billion to the trade deficit.

With the exception of General Motors Corp., which plans to import Japanese small cars as part of its fleet, the other three major American auto makers and the UAW wanted the quotas continued. Chrysler Corp. and the Ford Motor Co. both said they now will be forced for competitive reasons to increase their imports of Japanese-made cars -- nearly 300,000 in the case of Chrysler.

On Capitol Hill, the initial reaction went against the president's move, and he was accused of handing Japan the bonanza of free entry to the United States without gaining market-opening trade concessions in return. Sen John C. Danforth (R-Mo.), chairman of the Senate Finance Committee's trade panel, said the unilateral removal of auto quotas "ratifies the $37 billion trade deficit with Japan" and means U.S. markets are open to Japan "regardless of protectionism" practiced there.

Fitzwater said President Reagan's decision "not to ask them the Japanese to do anything" represents "an action that is deemed in the best national interest of the United States in light of the the improved performance of the U.S. auto industry," which now is able to compete in world markets.

Fitzwater refused to speculate on any action the White House might take on auto quotas or other trade retaliations if Japan refuses to come through with equal access to American products on telecommunications. Negotiations on that issue are under a March 31 deadline because of regulations that could either increase chances of U.S. sales or slam the door shut even tighter than it now is.

According to reports here, the talks on telecommunications and wood products are not going well.

Nonetheless, Commerce Undersecretary Lionel H. Olmer will go to Tokyo next week to continue the telecommunications talks now that the Japanese have decided to go into the specifics of their upcoming regulations, Commerce Secretary Baldrige said. He refused to allow Olmer to go to talks set for this week because he said the Japanese wanted to go over "fundamentals" that had been covered in a first round of talks.

"There was a commitment by Prime Minister Nakasone to open Japanese markets. We think he will carry that through," Baldrige said.