Two key members of the National Commission on Social Security said yesterday that President Reagan and House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) must participate in private negotiations or the panel will fail to propose solutions to the threatened bankruptcy of the system.

"We are now at a point," commission chairman Alan Greenspan said, "where I think it probably requires a judgment and agreement at the next level."

Appearing on "This Week With David Brinkley" (WJLA, ABC), Greenspan said, "Clearly, were the president and speaker to agree, not necessarily publicly, or even acquiesce in the nature of some general agreement, we probably could get virtually unanimous commission agreement on the very specifics of the resolution of the problems."

Using much harsher language directed more to Reagan than O'Neill, Sen. Robert J. Dole (R-Kan.), chairman of the Finance Committee and a commission member, said the White House has "been frightened to death by Social Security." Dole spoke on "Face the Nation" (WDVM, CBS).

Dole said the commission, which is to complete its work by the end of this week, has reached "the point where if he the president wants his commission to make any solid recommendations, he's going to have to say to the commission 'This is my commission and I want you to make recommendations.' "

Greenspan, who has been conducting preliminary, private discussions with White House officials, signaled some optimism over the prospect that the commission, which has been deadlocked, will resolve most of its differences and be able to make formal proposals to Congress.

"In the last week or two, I've sensed that there is significantly increased interest, essentially among the creators of this commission, to see whether the solution can be struck," he said.

Dole, however, was more pessimistic. "I doubt much will happen," he said, "unless the president decides to take a look at Social Security and become more actively involved. . . . Now the commission is about to expire, and there is still no one in the White House who is talking to the commission."

Last week, the commission, which had been scheduled to go out of business at that time, agreed to meet once more, after Greenspan reportedly told members that the White House has hinted at a willingness to enter into negotiations on remedying the system's problems. The "final" meeting is scheduled Friday.

The 15-member panel, composed of five persons appointed by Democratic congressional leaders and 10 by the president and Republican congressional leaders, has been bitterly divided. The Republicans have generally advocated reducing the system's projected deficits of $150 billion to $200 billion over the next seven years by cutting benefits, while Democrats have generally supported tax increases.

Neither approach has much political appeal, and one of the commission's original functions was to take the heat for initiating what everyone knew would be painful choices, particularly as the 1984 election looms.

Greenspan said the commission has succeeded in persuading Democrats and Republicans to agree on a much narrower range of possible solutions to problems of financing the system.

"We've ground the whole dialogue fairly substantially closer to solution," he said. But he acknowledged that there is no immediate prospect of Reagan and O'Neill beginning negotiations.

Dole said, "My view is, to be very candid about it, that unless the president signs off and the speaker signs off on any recommendations , I don't care what the commission does, we're just wasting our time."