The House strongly backed its own investigation of South Korean influence-buying yesterday by voting 321 to 46 to demand testimony from a former Korean ambassador suspected of making cash payoffs to members of Congress.

Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) made a rare floor speech in support of the nonbinding resolution which threatens that nonmilitary aid will be cut if the Koreans continue to balk at making diplomat Kim Dong Jo available for questioning under oath.

"We will not tolerate evasive responses in the face of such a serious matter," O'Neill said with emphasis, to applause from his colleagues. "Let there be no doubt that the consequences will be great if the Republic of Korea refuses to cooperate."

The South Korean embassy responded quickly yesterday that the House request was "unacceptable."

Spokesman Kim Su-doc said in a prepared statement that "we have cooperated to the fullest extent that comity, practice and the law of nations require of any self-respecting country . . .

"However, to allow former ambassador Kim to testify as requested in the resolution, under oath, affirmation or comparable means of assuring reliability is unacceptable."

The swift embassy response seems to leave little room for the two sides to work out a compromise, and makes a showdown likely in the next few weeks when a foreign aid appropriations bill comes to the House floor.

Senior State Department officials have opposed the resolution, saying coercing testimony from Kim would set a precedent that might be used against American diplomats. Kim, who was ambassador here from 1967 to 1973, is protected by diplomatic immunity from being compelled to testify.

Leon Jaworski, special counsel for the investigation by the House Committee on Standards of Official Conduct, has insisted Kim waive his immunity and tell committee investigators about cash payoffs he is believed to have made to as many as 10 current members of Congress.

O'Neill, who promised Jaworski his support when he urged the former Watergate special prosecutor to head the House inquiry last summer, noted yesterday that committee investigators have offered to take Kim's testimony overseas, through an intermediary, and to limit their questioning.

"Mr. Jaworski has conceded enough on the part of this House," the Speaker said.

O'Neill recounted the 54,000 lives and $11 billion that the United States has expended in South Korea in the past three decades."If this relationship is to continue," he said, the Seoul government must cooperate with the House.

He said he had emphasized to Korean officials that Congress is a "separate and equal branch" to deal with. This was an obvious reference to the opposite position the executive branch has taken on the issue.

Yesterday's action reverses a vote the House took three weeks ago in defeating an amendment by Rep. Bruce F. Caputo (R-N.Y.) to cut the fiscal 1979 budget resolution by the $56 million earmarked in Food for Peace aid to Korea.

The new resolution, had been supported unanimously by the leadership, the investigating committee and the International Relations Committee. It was approved by 225 Democrats and 96 Republicans, with 23 members of each party voting no.

The negative votes followed no ideological line. Several liberals as well as archconservatives opposed the measure.

Rep. Paul Findley (R-Ill.), who voted for the resolution in committee, led the opposition yesterday. He contended passage would hurt the United States more than South Korea because the resolution threatens $56 million in food sold by American farmers.

Rep. Henry B. Gonzalez (D-Tex.) said the resolution was "blackmail pure and simple. We would be honor-bound to cut off the aid." The choice was one of "humiliating the Koreans or of humiliating ourselves by failing to act," he said.

Rep. Samuel S. Stratton (D-N.Y.), a senior member of the Armed Services Committee, said passage would mean "we're going to move closer to war in Asia."

And Rep. B.F. Sisk (D-Calif.) criticized Jaworski, saying the "cloud of suspicion over the House" referred to so often in the debate might be only a "figment of the imagination of the staff man who has been riding this turkey."