The pressure of record trade deficits has spawned "a new toughness" among American business leaders, congressmen and administration officials toward Japan and other major U.S. trade competitors.

This deepening sense of frustration is emerging as the Reagan administration presses for a new round of international trade negotiations that would focus on areas such as services and high technology in which the United States is most competitive and feels it is unfairly treated.

In calling for the new talks in his State of the Union speech Wednesday night, the president said: "We have seen the benefits of free trade and lived through the disasters of protectionism."

But Reagan now faces new domestic problems from people who over the past four years supported his administration's espousal of a free-trade philosophy. Key leaders in business, Congress and the Reagan administration have begun adding their voices to those who believe that U.S. exports are not getting fair treatment abroad. While their foreign competitors find an essentially open market here, the American exporters face complex official and informal barriers in foreign markets, U.S. officials and business executives complain.

Most of this frustration focuses on Japan -- which last year accounted for $36.8 billion of the record $123.3 billion U.S. trade deficit worldwide -- but it spills over to Western Europe and newly industrialized nations such as Mexico, Brazil and Korea.

"There is a new toughness that has appeared among leaders of the business community," said Calvin J. Cohen, vice president of the Emergency Committee for American Trade (ECAT), formed to fight protectionist moves.

"Free trade is pretty much of a myth," Edward Donley, chairman of Air Products and Chemicals Inc. of Allentown, Pa., a major international manufacturer of industrial gases, said at a meeting of the Business-Higher Education Forum last month.

"In all areas of this town, including the administration, there is an increasing level of frustration and concern. Outside Washington, it's tangible," said U.S. Trade Representative William E. Brock, the White House point man on trade.

"There has been a qualitative shift in the attitudes of most of the business people who come in here," he continued. "It's not just overt protectionism. We are not hearing pleas that 'you have got to protect my particular business.' They think of it more on a national basis."

He said they don't believe negotiations will succeed in getting American exporters a fair shake in foreign markets. "They are coming to the conclusion that it's going to take a two-by-four," Brock added.

Many of the business leaders say they are not demanding protection, but rather a tougher policy of restricting access to American markets to countries that raise barriers to U.S. exports, in hopes of reducing those barriers.

What is not clear is whether the growing frustration over trade will simply stir up protectionism, or instead will help produce the successful new negotiations that the Reagan administration seeks to open trade channels wider.

Right now, only the anger is obvious.

The report last week of the $123.3 billion trade deficit for 1984 -- almost twice the previous record set in 1983 and three times the 1982 deficit -- sent explosive shocks through the business and political leadership.

Sen. John C. Danforth (R-Mo.), chairman of the Senate Finance Committee's trade panel, said the magnitude of the trade deficit triggered his decision to make an end to limits on Japanese auto imports in this country conditional on increased sales of American products in Japan.

The strong dollar, whose value has soared 40 percent since 1980, bears major blame for the trade deficit by raising the price of U.S. exports overseas and making imports less expensive in this country, trade officials and business leaders acknowledge. Nonetheless, they believe substantial cuts could be made in the deficit if American companies had equal access in foreign markets.

For example, the administration estimates that about $10 billion in highly competitive American products are kept from the Japanese market by tariff and nontariff barriers.

On Capitol Hill, Danforth, who is considered a free-trader, described himself as "exasperated" and "fed up" with the United States' trade relationship with the Japanese, which is "all give on our part and all take on their part."

"Negotiating with Japan is a waste of time. It's trying to talk them into doing something they are just not going to do," he said in an interview.

The same concerns are boiling up within the National Association of Manufacturers, said Jerry J. Jasinowski, senior vice president and chief economist. "It's reached a point of outrage. Business leaders across the spectrum regard the trade deficit and the problems that lead to it as on a parallel with the budget deficit," he said.

"It's pretty much across the board. It reflects the fact that we have a pretty healthy domestic recovery, at the same time that firms have made themselves extraordinarily more efficient, and yet we're losing markets," he said.

The trade deficit, moreover, "is radically changing the way American firms are doing business, driving more of them abroad and increasing the amount of outsourcing buying components overseas ," Jasinowski said.

Although there was no policy statement from the forum meeting last month, strong sentiment came from the leaders of more than 60 major corporations and universities for a tougher U.S. trade policy. Access to American markets should be used as a lever to gain sales for U.S. exports overseas, said the Rev. Theodore M. Hesburgh, president of the University of Notre Dame and chairman of the forum.

Brock said this sharp swing against free trade among business leaders and lawmakers will affect the Reagan administration's trade policy.

The first signs of a shift within the administration came during the internal debate that preceded the Los Angeles meeting a month ago of Prime Minister Yasuhiro Nakasone with President Reagan, where officials of the traditionally antiprotectionist Treasury Department sided with Brock and Commerce Department Secretary Malcolm Baldrige in urging a tougher stance on trade issues with Japan.

The experience of negotiating with Japan over the opening of its financial markets turned the Treasury leadership around, administration trade sources said. The State Department traditionally has shielded Japan from trade pressures out of concern for the broader U.S.-Japanese strategic relationship, but there are indications that this, too, is changing.

Japan, however, has appeared to be taking a hard line in recent follow-up talks in Tokyo to the Reagan-Nakasone talks.

Despite President Reagan's insistance that American companies be given the same right to sell in Japan that Japanese companies have in America, Commerce Undersecretary Lionel Olmer returned from talks in Tokyo with the belief that the changes now under way in Japan's telecommunications system "do not contemplate carrying a lot of foreign baggage."

"We don't want another trade package," said Olmer, who traditionally has supported tough stances on trade with Japan. "We want that market opened from top to bottom, just as ours is."

Japan's astute ambassador to the United States, Yoshio Okawara, stressed the emergence of the fairness issue in a New York speech to Japanese businessmen last month. "It is important that Japan understands this idea of fairness," the Japanese Embassy quoted Okawara as saying.

But Brock, who has spent much of the past four years trying to deflect protectionist pressure from American industries, warned against lashing out in ways that will hurt the United States more than its trading partners.

"The trouble is when you get into these moods, it's tempting to do something, anything, to demonstrate you are trying. With that kind of climate, it's easy to do the wrong thing," he said.