House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) is under pressure to ease into the background during his last two years in Congress and allow younger Democrats to craft a new image for the party in the wake of President Reagan's reelection landslide.

According to leadership aides, O'Neill is open to some of the changes recommended by the young Democrats, including giving them a greater role in speaking for the party on policy issues.

At the same time, Democratic conservatives, particularly from southern states where the Republicans have made significant gains, are demanding that O'Neill step down as speaker. Rep. Charles W. Stenholm (D-Tex.) said a small group of "Boll Weevil" conservatives may run a challenger to O'Neill when the Democratic Caucus meets here Dec. 3 to select leaders for the 99th Congress.

Grumbling about the House leadership is normal whenever Democrats lose an election, and this year's soul-searching is not likely to force any dramatic changes in the leadership.

But lawmakers said this week that the sentiments are more widespread than in the past, prodded in part by the national debate about the future of the party and by worries that Democrats in Congress and elsewhere could face even more severe electoral problems in 1986.

The pressure for change also is a first round in the House leadership battles expected in 1986, when O'Neill has said he will retire, putting the speaker's position and the leadership jobs under it up for grabs.

While House Majority Leader James C. Wright Jr. (D-Tex.) is O'Neill's heir apparent, there have been rumblings, particularly among some younger Democrats pressing for changes this year, of potential challenges to him or to Majority Whip Thomas S. Foley (D-Wash.), who would like to move up to majority leader.

The strongest pressure on O'Neill and the leadership is coming from a group of 25 to 30 younger, increasingly influential Democratic lawmakers, elected within the last 10 years, who have felt frustrated by their lack of influence on leadership decisions.

The group, formed by Rep. Tony Coelho (D-Calif.), head of the House Democratic campaign committee, and Rep. Richard A. Gephardt (D-Mo.), met privately this fall to voice what several participants said was a strong concern at the leadership's failure to develop a clear Democratic legislative agenda.

"A number of us felt it would be a good idea to reflect on . . . how the whole operation in the House could be improved," Gephardt said.

According to these lawmakers, sentiment was developing before the election for a coup to unseat O'Neill, but the talk abated when Democratic losses in the House last week were kept to 14.

In meetings with O'Neill and Wright this week, Coelho, Gephardt and other members of the rump group told the speaker, who will turn 72 next month, that there was strong sentiment that control of the House must include some of the younger Democrats.

"There is a strong feeling that it is necessary to broaden the public picture of the party on the public platform in a way that can give the party a different image," said one Democrat, who spoke on the condition he not be identified.

While most members of this group said they have strong personal affection for O'Neill, they view him as a relic of the party's past, out of touch with the voting mainstream.

The rump group also has asked O'Neill to set up a speaker's cabinet or executive council, including some younger Democrats, to help structure the House's agenda, act as a better liaison with the Democratic caucus and develop broad themes for the House.

Rep. Timothy E. Wirth (D-Colo.), who attended some of the rump group's meetings, said members thought the House leadership in Reagan's first term had "not been very creative" in articulating a vision through legislation, tending instead "to bring up whatever comes out of the committee structure. It follows no policy."

The demands reflect the belief that the leadership spent too much time last year reacting to Reagan and not enough time creating a consensus on issues important to Democratic members.

Leadership aides said it is likely that O'Neill will agree to broaden leadership meetings, hold more caucus meetings and occasionally step back to let some of the younger Democrats speak for the party in Congress.