Frustration is rising in Congress over the political and military stalemate in El Salvador, increasing the pressure on the Reagan administration to insist on talks between the guerrillas and the Salvadoran government.

Key members of the Senate Foreign Relations and House Foreign Affairs committees say Congress may attach new conditions to the administration's 1984 military-aid request for El Salvador, perhaps requiring that the Salvadoran government initiate talks with the leftist opposition in order to receive U.S. assistance.

"We need to find some framework for talks with the guerrillas," said Sen. Nancy Landon Kassebaum (R-Kan.), a moderate on Foreign Relations. "The administration should make this the focus of their policy. We've always said a political solution is the only viable solution, but we're being pushed into a military solution."

Kassebaum said she would not support the administration's request for an increase in Salvadoran military aid from $26.3 million this fiscal year to $86.3 in fiscal 1984.

"We'll see a real fight," she said. "I don't think Congress will go along with it unless the Salvadoran government shows reforms in their judicial system, better control of their security forces, a willingness to move toward talks and a reduction of regional tensions."

The tone of congressional Democrats, who strengthened their numbers in the House in the November election, is even more impatient.

"There's a mood of incipient insurrection in Congress," said Rep. Stephen J. Solarz (D-N.Y.). "There's a growing concern our policy is leading nowhere."

A six-page "Democratic Alternative" issued Dec. 9 by a group including Rep. Michael D. Barnes (D-Md.), chairman of the House subcommittee on Western Hemisphere affairs, and Sen. Alan Cranston (D-Calif.), a ranking member of Foreign Relations, said, "The Reagan administration has chosen a policy which lends confidence to the terrorist right; which motivates and strengthens the terrorist left; which contributes in doing so to further violence; and which has provided nothing but bitter discouragement for those genuinely committed to peace and democracy in the region."

The document calls for a halt in what it alleges is administration support for a covert war against Nicaragua, which, it said, has produced "a propaganda bonanza for the radical left, an excuse for a military buildup by Managua and a growing threat of regional war."

Urging the administration to join with Mexico and Venezuela to initiate discussions with Honduras, Nicaragua and El Salvador, the group said the administration should support negotiations with Salvadoran guerrillas "without preconditions."

Last fall, the Salvadoran government refused to negotiate with guerrillas until they lay down their arms. Assistant Secretary of State Thomas O. Enders last week brushed aside suggestions of talks, telling Barnes' subcommittee that the Marxist guerrillas are not interested in participating in a democratic government.

"The mood in Congress is stronger for a change in policy than it was six months ago," Barnes said. "My colleagues are all coming up to me in the halls and saying, 'What the hell are we going to do about El Salvador?' There's a real frustration that we're floundering. The euphoria of the Salvadoran elections has worn off. There's the sense in Congress that the war is not going anywhere from a military perspective, and the political situation is chaos."

Eighty House members are co-sponsoring a bill introduced by Rep. Gerry Studds (D-Mass.) to declare "null and void" Reagan's certification that El Salvador has been making progress in human rights, economic reforms and political democracy. The certification is required twice a year for the country to be eligible for U.S. military aid.

While the Studds bill, which would suspend military aid, has the strong support of church groups and, recently, of the AFL-CIO, it appears unlikely to pass. Members fear that an abrupt aid cutoff would play into the hands of extremists on the right and the left, while leaving the U.S.-supported moderates with no power base.

However, sentiment for stricter certification provisions is widespread. Solarz, who visited El Salvador last month, plans to lead an effort to rewrite the certification legislation to halt military aid unless the Salvadoran government agrees to an unconditional dialogue with the opposition.

Solarz also would make certification conditional on a halt to the murders of civilians by death squads. "Five to six thousand deaths a year is still too many," he said. "If Guatemala eliminated the death squads, El Salvador can too."

Under current law, the Salvadoran government must show only that it is "achieving substantial control" over its security forces, which are widely blamed for the free-lance killing.

Under Solarz' plan, Congress would be able to veto the president's certification under procedures that would guarantee floor votes.

Moderate Republicans fear the political consequences of continued stalemate. "If the administration can pull off a low-key negotiated settlement, it will be a measure of Reagan's success in foreign policy and a political plus," said Rep. Jim Leach (R-Iowa). "If the stalemate continues, it will be a campaign issue for the Democrats in 1984."

Leach, who traveled to El Salvador with Solarz, wants the administration to open talks with Nicaragua and to deal with El Salvador through a special U.S. envoy.

Such a move would be opposed by the right wing of the Republican Party.

The frustration of many members of Congress was expressed by Rep. Peter Kostmayer (D-Pa.), who told Enders last week that he agreed it would probably take a long time to lead El Salvador into democracy, and continued military aid ight be necessary.

But Kostmayer, who said his mail has been running "100 percent against aid," noted that "the levels of political tolerance in our country are declining . . . . I find that, with all due respect to my own constituents, that their opinions are somewhat uninformed on the matter and that they are quite intolerant of what is happening" in El Salvador.