Congress may be forced into a trade war involving the adoption of openly protectionist measures because of the "failure" of high-level international trade talks in Geneva last week and continuing high unemployment levels here, Senate Finance Committee Chairman Robert Dole (R-Kan.) warned yesterday.

"The climate is now right for protectionist measures in Congress," said Dole, a delegate to the recently concluded meeting on the General Agreement on Tariffs and Trade.

Moreover, he added: "I am not prepared to say that, given the current situation, selective protectionist measures are not the right approach."

Referring to European agricultural subsidies that the Common Market had refused to adjust at the GATT session, Dole snapped:

"I hope we don't get in a big trade war, but we'll be under a lot of pressure to teach somebody a lesson, or show that we mean business.

"We can't stand by and see our farmers disappear, so, while no one wants to start a trade war with your tax dollars [used as subsidies], I'm not sure we can avoid it," Dole said in a luncheon address to a trade issues seminar sponsored by the American Enterprise Institute.

Meanwhile, Sir Roy Denman, trade negotiator for the Common Market, told reporters that "no one is going to rewrite the principles of the [European Community's] common agricultural policy any more than anyone is going to rewrite the U.S. Constitution."

Denman reiterated that Europe would continue to resist American pressures to cut subsidies to its farmers. "The question [for the Europeans] is: 'Are we going to throw another million farmers on the scrap heap to please the folks in Iowa?' " Denman said.

In a separate address to the AEI session, U.S. Trade Representative William Brock said that "whether or not we can avoid a trade war will depend on whether we can make some progress bilaterally with Europe."

Dole contended that record American trade deficits, including huge imbalances with Japan, are "indicative to many members of Congress and their constituents of the failure of the multilateral system. The average American simply doesn't understand why Japanese cars and TVs are sold here, but U.S. cigarettes and baseball bats cannot be sold in Japan."

He noted that the political pressures for protectionism might well prove irresistible, observing that former vice president Walter Mondale and Sen. Edward Kennedy (D-Mass.) are now "ardent supporters of the local-content concept, notwithstanding the drastic consequences it implies."

The local-content bill, which would require all cars sold here to have a high content of American parts and labor effort, has been marked up by the House Commerce Committee, and is now on the House calendar. This bill, which has been denounced by Brock as "the worst trade legislation in many years," could pass in the lame-duck session. Even if it doesn't, "It clearly will be an issue for some time to come," Dole said.

In response to questions, Dole readily conceded that his preferred position favoring free trade was being compromised by the critical question of jobs. "We're up against economic and political realities," Dole said. "A lot of people are out of work, and we're all looking for a solution."

Brock made plain his disappointment in the limited accomplishments of the GATT meeting, which he rated no better than a 'C' -- a grade he said could drop to an 'F' if the signatories fail to follow through on the extremely vague language of the communique.

Brock said that the GATT meeting had only barely avoided "tragedy," and at the last minute had managed only "a limited movement in the right direction.

"Maybe we got about half of what we would have liked [in the areas of services, investment, and high technology combined], but we got very, very fuzzy words, especially on services," Brock said. He noted, however, that the Third World effort to have the services question moved from GATT to the U.N. Conference on Trade and Development (UNCTAD) had been resisted successfully. No one wants to see "the Unctadization of GATT," he added.

He warned that "the year 1983 may be the toughest trade year we've seen in a long time," because there will be pressure on many nations to take protectionist actions to counter import competition.