Reagan administration officials ended two days of trade talks here today by issuing an ultimatum to Japan to move quickly to open its markets to more American goods or face a protectionist backlash in Congress.

Economic tension between the two countries has escalated in the wake of the United States' huge and sharply expanding trade deficit with Japan, a figure which now is expected to balloon beyond previous estimates of a record $15 billion this year to between $18 billion and $19 billion, according to senior American officials.

Deputy U.S. Trade Representative David Macdonald, who led the American negotiating team, told reporters today, "Japan must make a decision, which we cannot make for it, whether to open its own market or to sacrifice some of its access to the U.S. market." He warned that economic relations between the two countries had reached "a critical decision point."

The Reagan administration's bid to pry open what U.S. officials generally regard as Japan's clamlike market is the latest of numerous similar attempts by the American trade delegations that have flocked here in recent years.

But despite indications that the Japanese government is beginning to take Washington at its word, trade analysts and officials here still doubt that there will be any immediate or significant turnaround in Japan's lopsided trade advantage.

In wide-ranging talks with Japanese counterparts, U.S. trade officials raised a number of specific issues, including what they view as overly stringent customs, product standard and testing requirements which foreign business executives frequently claim bar them from successfully competing in Japan. They also pressed demands for the early removal of Japanese quotas on imports of American cigarettes, beef and citrus fruit, among other items.

Macdonald characterized the talks as "a constructive beginning," but warned Japan that it must take "prompt, effective action" to induce the flow of more American products. Citing high levels of unemployment in the United States and the "great frustration of American businessmen trying to penetrate and sell manufactured products in Japan," he said "we must be on the way to a solution quickly in order to avoid a political problem in the United States."

Asked what measures Congress might take if Japan failed to respond, Macdonald pointed to the proposed revision of the U.S. Telecommunications Act that would bar products from countries not opening their markets to similar American goods.

The legislation has cleared the Senate and is now before the House. "This is one of the illustrations we gave to the Japanese side," he said, suggesting that similar steps calling for reciprocity in trade potentially might encompass other products.