Prime Minister Yasuhiro Nakasone, seeking to avert a protectionist backlash in the United States, today ordered his Cabinet to work out a new set of steps to reduce Japan's trade barriers and open its markets to more American goods before he goes to Washington in January for talks with President Reagan.

Nakasone's move came as Japan and the United States ended a series of high-level trade talks here today during which Reagan administration negotiators have hinted broadly that relatiatory measures might be taken by early next year unless Tokyo promises to act on a raft of long-pending trade complaints.

These include U.S. demands that Japan open its doors to more agricultural imports -- chiefly beef and citrus fruit -- and streamline its customs and product standards procedures, which American business executives complain bar them from competing effectively in the market here.

Apparently reflecting the top priority the new premier has put on improving badly strained economic ties with the United States, he told his cabinet that "resolving trade friction with the U.S. and European countries is the government's greatest concern," according to reports in the Japanese press.

Although Nakasone, who was named prime minister Nov. 26, stepped back from specifying what the new round of trade reforms might contain, he is widely expected to strike a clearer stand on trade policy than his predecessor, Zenko Suzuki. Analysts here said, however, that it remains to be seen whether Nakasone will be any more successful in making headway against the entrenched interest groups in industry and government here that strongly have opposed the sweeping changes that the United States wants.

At today's Cabinet meeting, Nakasone stressed the need for further trade concessions in advance of talks at the White House on Jan. 18. Economic tension between the two countries has escalated in the wake of the United States' huge and sharply expanding trade deficit with Japan, which is expected to hit a record $20 billion this year. Two previous packages of market-opening measures announced by Tokyo earlier this year have fallen far short of U.S. expectations amid mounting calls in Congress for so-called reciprocity trade penalties against Japan.