The House yesterday rebuffed the leadership of both parties and balked at lifting the limit on members' outside income, a limit fixed in a major ethics reform act four years ago.

By a vote of 271 to 146, members defeated a leadership-backed resolution that would have let them earn up to 40 percent of their salaries from outside sources. The current ceiling, established in 1977, is 15 percent.

Opponents warned that lifting the ceiling would open the door to abuses and conflicts of interest of the sort that the House had tried to end in the post-Watergate era of congressional reform.

They also argued that voters next year would react strongly to an effort by members to increase their own income at a time when they were also cutting back federal social programs benefitting others.

The House was jittery about voting itself an outside-income favor in the wake of action earlier this month giving members of both the House and Senate bigger tax deductions to cover expenses of living in Washington. The Senate in addition refused to impose on itself the 15 percent ceiling on outside earnings and now operates without any limits.

In the past, some lawmakers have earned large incomes in the form of fees to their law firms or honoraria for speeches to private groups. The practices were substantially curtailed in the House in 1977. The Senate has never gone along with any limits, despite an agreement in 1977 with House leaders to impose a similar 15 percent restriction.

A study by Common Cause showed that few House members exceed the limits now in effect, which prohibit earning more than $9,099 in addition to official salaries. In 1980, the lobby group reported, only 17 percent of the 356 current members covered by the limit that year earned more than $8,000 from outside sources. About two-thirds of those members had outside income of less than $5,000 each.

Some House members had advocated removing the ceiling entirely to match the Senate, but House leaders settled on a compromise of 40 percent, which would have enabled each member to earn up to $24,264.

House Speaker Thomas P. O'Neill Jr. (D-Mass.) told reporters yesterday morning it was "ridiculous" for House members to treat themselves differently from senators and observed that some members have been forced to give up their outside businesses to serve in the House. His sentiments were echoed on the House floor by Rep. Trent Lott (R-Miss.), the minority whip.

But their arguments were overridden in a rash of floor speeches that warned against rolling back the post-Watergate ethics reforms. Rep. Lee H. Hamilton (D-Ind.) claimed it would revive the specter of conflicts of interest and make it appear that congressmen are "cashing in on their congressional position."

Against those warnings, several House members contended that limits on outside income are tending to produce a Congress in which only the rich can afford to serve. "A growing percentage of the House members are millionaires," said Rep. Paul Simon (D-Ill.).

All Maryland members voted against lifting the ceiling except Reps. Steny Hoyer and Parren J. Mitchell, Democrats.

Among Virginia congressmen, Reps. Paul S. Trible Jr., G. William Whitehurst, M. Caldwell Butler, Stanford E. Parris and Frank R. Wolf, all Republicans, voted against the measure. Democratic Rep. Dan Daniel and Republican Reps. Robert W. Daniel Jr., J. Kenneth Robinson and William C. Wampler voted for it. Rep. Thomas J. Bliley Jr., a Republican, did not vote.