Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) yesterday ruled out House action on a controversial instant-voter registration bill before next year, and said another measure to create a federal consumer protection agency also is virtually dead for this session.

Both bills were up against major opposition in the House, and even if passed, they were almost certain to face a fillbuster in the Senate.

The voter registration plan - to let people register at the polls on election day beginning in 1978 - was the key proposal in an election revision package that Sen. Edward M. Kennedy (D-Mass.) offered, and it got strong White House backing.

Under the consumer protection legislation, a new independent agency would be created to represent consumer interests in cases before other federal agencies.

In telling reporters that the voter-registration plan would not come up for House consideration before adjornment this year despite severa compromises, O'Neill said, "We had a lot of agreements, but it fell flat on its face."

At one point, O'Neill, Vice President Mondale and others had worked out a modification of the bill to make the plan optional for the states.

O'Neill said yesterday that he still favored the concept but that it would be better to "start from scratch" and "make an all-out pitch" for a new bill in January in view of "so much criticism" of the pending legislation.