An effort is under way, pushed by younger House Democrats, to change the House rules to prohibit riders, such as the controversial abortion amendment, from being attached to appropriations bills.

Although the House Democratic leadership is not behind the project yet, Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) and others have talked about finding some way to avoid tying up fiscal 1979 funding bills with essentially legislative amendments.

The Democratic Study Group staff is also looking into the situation.

Last year the abortion language, which sought to ban the use of Medicaid funds for abortions, held up the Labor and Health, Education and Welfare money measure for five months and required 11 roll-call votes on the House floor before it was approved.

In June, House Republicans offered several amendments to appropriations that forced votes on veteran pensions for Vietnam deserters and homosexual rights. The abortion language materialized in July.

Since that time, according to a DSG official, "the idea has been bubbling and now must be taken seriously."

O'Neill, according to Democratic aides, has discussed more than once since last summer finding a way to limit money bill amendments.

A main problem for the leadership, according to House aides, "is whether a rational ground for doing something can be found."

"It is too early to tell whether anyone can come up with something," an O'Neill aide said yesterday.

Rep. Herbert E. Harris II (D-Va.), a second-term member, said yesterday he will circulate a proposed rule change next week.

His plan would sharply restrict money bill amendments to those that "reduce or increase an amount of money included in the bill." It would apply both to amendments in the House Appropriations Committee and offered on the House floor.

Harris said yesterday he had talked to a number of his colleagues and was "surprised at the number who said yes" to his approach.

House rules have always prohibited appropriations bills as legislative vehicles. But under an 1876 precedent, members have been able to offer language from the floor that relates to how those funds can be spent. Called the "retrenchment loophole," it was originally supposed to be limited to explaining where or how cuts voted in expenditures should be made.

Expanding on that over the years, members have designed a variety of approaches that Harris says add up to the very legislation that the rules now are supposed to prohibit.

Harris said, "I don't think the House looks worse than when it deals with amendments in the [money bill] framework . . . it is a way to take a highly emotional issue and a political thing on the floor without a hearing or something substantive done in committee."

Not everyone, however, agrees with Harris on the matter.

An O'Neill aide, for example, pointed out yesterday that sometimes amending an appropriations bill "may be the only annual forum for some issues."

Often cited in this context were the defense appropriations bill amendments during the Vietnam war period that became the vehicles for showing congressional opposition to the U.S. policy.

Harris, however, said yesterday "There was another way to have handled that . . . There was a way to do it if you had the votes."

Harris, in his proposal, plans to allow only one form of amendment that goes beyond just numbers. It would apply when the pertinent legislative committee votes to authorize a member to offer such an amendment.

Thus, for example, on the abortion amendment, the language would only have been in order if either the Ways and Means Committee or the House Interstate and Foreign Commerce Committee, both of which hve authority over the medicaid program, had approved.

Rep. Bill Frenzel (R-Minn.), a member of the Rules Committee, which would handle any proposed change, took a critical view yesterday of the Harris idea.

"Unfortunately, whenever our young Democratic colleagues disagree with a proposal," Frenzel said acidly, "they want to change the rules."

Harris says he plans to see what response he gets to his present draft and then offer his amendment to the Rules Committee.

That body, which approves House rules at the start of each session, is traditionally reluctant to change things in mid-session - the situation that prevails today.

An O'Neill aide sees the possibility that the matter will be confronted in June, when the fiscal 1976 Labor-HEW or some other appropriation bill is set to come on the House floor.

"At that time we may see an effort to prohibit or restrict what amendments can be offered," he said.