Rep. Thomas P. (Tip) O'Neill Jr. (D-Mass.) wound up his first two years as Speaker as he began them - a strong leader, immensely popular with his colleagues, but caught in the middle of an uneasy relationship between Congress and President Carter.

O'Neill wins high marks from his colleagues across the political spectrum for his decisiveness and his ability to make an unusually fractious, independent and unsettled House move on important legislation.

For instance, O'Neill's move early in 1977 to put warring committees with energy jurisdiction under the umbrella of a single ad hoc energy committee, is widely credited as being the main reason that the House was able to produce an energy bill at all.

O'Neill's insistence that what was actually five bills be presented as a single bill with only one vote allowed was crucial to its passage.

"O'Neill is the best Speaker since Sam Rayburn and he may be better than Rayburn because Rayburn could win intimidation where O'Neill must do it all through persuasion, one senior Democrat said.

But unlike his counterpart, Senate Majority Leader Robert C. Byrd (D-W. Va.), O'Neill also decided to take on the role of bridging the gap between the Congress and the "outsider" who was president, playing friend and "Dutch uncle" in trying to explain the ways of Congress to Carter, and actively working for Carter's programs on Capitol Hill. There were quiet dinners with the Carter family and long private chats in which O'Neill expounded and Carter listened to stories of the workings of the Hill.

Now the problem is that the White House may have learned the lesson too well, putting O'Neill on the spot.

The White House showed in the closing days of Congress it has learned how to wheel and deal for the bills it wants. "They put all the chips on the table for energy and public 'works," a Hill source said.

The Question is how Carter will use his new-found power. "Tip hopes he will use it positively rather than negatively." said O'Neill aide.

In addition to preaching to Carter that consideration of what members of Congress want may be important to winning votes, O'Neill urged Carter to avoid confrontations with Congress. "If he wants to be a great president, he will have to have the support of Congress," O'Neill said at one point.

But in Carter's view, avoiding confrontations with Congress didn't work. The more he did, the more he lost - substantive parts of his energy and tax bills, a bill to control hospital costs - and the more his image weakened until the question of his ability to deal with Congress, his "competency," became an issue.

Carter turned around that image by confronting Congress - successfully vetoing a defense bill with a nuclear carrier he didn't want and vetoing a public works bill with water projects he considered wasteful bonndoggles.

O'Neill problem is that despite Carter's post-Camp David image, there are still may senior House Demorats who feel alienated from the present White House on both substance and style, and many junior House Democrats who pride themselves on their independence from both the president and the Democratic leadership.

If Carter's mind is set on more confrontations, O'Neill may be forced to choose between his loyalty to the president and to his colleagues, many of whom were telling him he was too close to the president.

A leadership aide said the White House would have to be "highly selective" in how often it pulled out all the stops and for what. "Do it often and these boys will put a price on everything," he said.

O'Neill feels his first Congress was highly successful, citing the energy bill, tax cut, reducing unemployment and reducing the budget below even what Carter wanted.

He expects his second Congress to put further emphasis on controlling inflation and cutting spending, overseeing the programs now in place.

And he hopes Carter has found a way of dealing with Congress that is neither too placating nor too antagonstic.