The Maryland House of Delegates voted 69-to-64 today against limiting state funding of abortions, as anti-abortion legislators and lobbyists split over what the sponsor of the legislation called the extreme tactics of one right-wing group.

Del. Timothy F. Maloney (D-Prince George's), who sponsored the narrowly defeated proposal to ban state-funded abortions carried out for reasons of mental health, said he was "very dismayed by some of the tactics" used by the Family Protection Lobby, which circulated brochures and advertisements charging that five delegates had voted "to use state taxes to kill innocent babies."

Maloney said the lobby's approach "really dampened my enthusiasm because I didn't want to be associated with some of their tactics. These are extreme tactics."

The Senate defeated a similar measure by one vote earlier this month.

The outspoken criticism of the lobby, a coalition of fundamentalist Christian organizations, reflects a deepening split among groups that seek to curb Medicaid funding of abortions.

Until this year, the leading anti-abortion group in the state was Maryland Right to Life, whose quieter lobbying efforts had met with some degree of success. Although still active, Right to Life has been displaced, at least in visibility and outspokenness, by the Family Protection Lobby.

The latter group, which lost its status last year as the Maryland chapter of Moral Majority, distributed brochures in the home districts of five delegates charging: "16,500 babies have died as a result of this man's vote!. . . If the use of state funds to murder helpless babies offends you, give delegate's name a call. Tell him to vote against the use of state taxes to kill innocent children."

The brochure asks for $5 contributions. The group also has sought to place advertisements in local newspapers. At least one newspaper refused to run the ads.

Jim Wright, director of the militant anti-abortion lobby, defended his group's tactics.

"We have always been factual and moved within the realm of propriety," he said today.

Wright led a demonstration earlier this week of some 500 persons who clogged the foyer of the State House, kneeling to pray for pro-abortion activists who, they say, do not "see the light." Today, members of his group, wearing bright yellow "Support Life" buttons, lay down in the corridors of a House office building to draw the attention of legislators who were preparing to vote on the abortion issue.

Del. Gerald Winegrad (D-Annapolis), one of the five targeted delegates, said that he had received numerous phone calls from constituents who were outraged by Wright's "Hitlerite, fear-mongering" tactics.

The others targeted, all Democrats, were Dels. Frank C. Robey and Dennis C. McCoy, of Baltimore, and Dels. Thomas B. Kernan and Donald K. Hughes of Baltimore County.

Wright has been at the center of controversy before. It was his unsuccessful attempt to close a Main Street bakery here for selling "X-rated" male and female gingerbread cookies that resulted in Moral Majority's breaking its ties with his group.

The House vote today was characteristic of the legislature's traditional stand on abortion funding. Although anti-abortion legislators have failed each year to abolish state funding of abortions, they have succeeded in restricting some instances where funds can be used.

The most significant gain came two years ago, when language was inserted in the budget saying that a woman must be in danger of severe and long-lasting physical or mental problems in order to justify an abortion.

June McGucken, an official with Maryland Planned Parenthood, said today that as a result of these stringent restrictions fewer abortions have been available to poor women in the state.

State health department officials said that in fiscal 1980, state and federal funds paid for some 6,500 abortions, compared to a projected 3,000 in fiscal 1982, all of which will be paid for by the state because of new federal restrictions that limit funds for abortions to where the mother's life is in danger.