Reagan administration trade negotiators today pressed Japan to make further efforts to open its markets to more American goods, indicating that a more sweeping round of trade measures would be required to head off protectionist legislation now before the U.S. Congress.

At the end of two days of high-level trade talks here, U.S. Deputy Special Trade Negotiator David R. Macdonald told reporters, "There is some likelihood of substantial movement by the Japanese government in connection with access to its market. If indeed this comes to fruition, I believe it would have a substantial effect on reducing or eliminating the rationale for protectionist legislation."

But the latest round of negotiations, aimed at easing badly frayed trade ties between the two countries, produced little in the way of concrete steps toward Washington's objectives of getting Tokyo to abandon quota restrictions on agricultural imports and dismantle a host of other trade barriers that American officials claim help bar competitive U.S. products from the Japanese market.

During the first day of talks Tuesday, Japan agreed to a U.S. proposal to set up a new working-level group that is scheduled to begin deliberations in Washington next month on Tokyo's 22 remaining quota restrictions on farm products, something that Washington views as symbolic of the closed nature of the Japanese market.

Tokyo also acceded to a longstanding U.S. demand to begin specific negotiations on Japanese imports of beef and citrus fruit in October, six months in advance of the schedule set out under the 1979 agreement between the two governments. The two sides also agreed to start talks on possible joint research and development projects in high-technology goods.

At the same time, however, U.S. trade officials acknowledged that they failed to obtain specific commitments from the Japanese on a list they presented aimed at freeing market access in 12 product areas, including nuclear energy equipment, computers, pulp and paper, soda ash, medical equipment, cosmetics, tobacco, wood products, telecommunications equipment and heavy electrical machinery.

At the outset of talks here, Macdonald reportedly told his Japanese counterparts that the U.S. trade deficit with Japan--a record $18 billion in 1981--could be cut by $5 billion to $10 billion if the Japanese allowed such commodities to enter their market more freely. But he acknowledged today that it would be "impossible" to predict what actual effect such liberalization might have on the trade imbalance.

The negotiations took place amid growing protectionist sentiment in the United States which has been fueled by rising unemployment and an anticipated trade deficit with Japan in 1982 that, according to American trade officials, could reach $25 billion. The grim outlook for two-way trade has prompted a number of "reciprocity" bills in Congress calling for equal access to the markets of trading partners.

The Japanese government has moved to avert the enactment of such legislation by pledging to speed up tariff cuts on 1,650 items in advance of the schedule agreed upon under the Tokyo round of multilateral trade negotiations. In late January, it announced the immediate easing or elimination of so-called nontariff trade barriers on 67 items compiled from a list of specific complaints by foreign traders and has promised another round of similar trade measures before the Paris economic summit in June.

But these measures have fallen considerably short of the expectations of top Reagan administration trade officials, and the Japanese have become increasingly irritated with what they perceive as Washington's lack of appreciation for their efforts.

Reflecting an apparent attempt to cool Japanese tempers, American trade officials said they believe the political commitment by the government of Prime Minister Zenko Suzuki is sincere: "We came away with the feeling that it has begun to spread further down in the bureaucratic leadership. The political commitment appears to be taking hold."

But they stressed that further efforts would be required on Tokyo's part, especially in the areas of streamlining customs practices, product standards and testing requirements and the opening of markets for more high-technology goods and service industries such as banking, insurance and finance.

Macdonald said, "I believe the Japanese government has made substantial progress in a number of service areas, particularly in the financial field. But we do not think that the Japanese government has fully opened all service areas as widely as the U.S." He added that the United States is considering raising the issue of trade in service industries at the ministerial meeting of the General Agreement on Trade and Tariffs in November.

Meanwhile, Hiroshi Fukada, Macdonald's Japanese counterpart, told reporters that the Japanese were concerned with the continuing high level of interest rates in the United States, which are widely viewed here as having aggravated the U.S. trade deficit by, in effect, lowering the dollar-denominated prices of Japanese exports.

Fukada also expressed Japan's interest in the "desirability of future American exports of Alaskan crude oil." Congressional regulation now rules out foreign purchases of Alaskan oil, but officials in Tokyo contend that potential imports by Japan, if approved, could help substantially to reduce the bilateral trade imbalance.