A group of conservative Maryland Democrats, hoping to spoil the 1986 U.S. Senate election for Republican incumbent Charles McC. Mathias Jr. and his possible Democratic rival, Gov. Harry Hughes, are planning to launch a long-shot independent campaign for Mathias' seat.

The group, led by former lieutenant governor Samuel W. Bogley III and other antiabortion activists in the Washington and Baltimore suburbs, hopes to capitalize on the liberal records of Mathias and Hughes on abortion and other politically potent issues, according to spokesmen.

"It's up to us to put forth a candidate . . . who will show how much alike Harry Hughes and Mac Mathias are on key issues," said Bogley, who indicated a desire to be the group's candidate next year.

A return to statewide politics would mark a change for Bogley, a Prince George's County resident who has maintained a relatively low profile since Hughes dumped him as lieutenant governor in 1982.

In 1978, Bogley was Hughes' last-minute choice as a running mate when Hughes was still a dark-horse candidate for governor. Later, after they took office, the two had a public falling out as Hughes took a liberal approach on many social issues -- including the emotional one of abortion -- while Bogley and his wife, Rita, championed the antiabortion forces in the state.

In an interview earlier this week, Bogley stressed that his involvement with a group of what he dubbed "independent Democrats" in no way represented a vendetta against Hughes.

Instead, organizers of the nascent campaign, scheduled to start officially later this month, want a Senate candidate who subscribes to the historic tenets of the Democratic party but who also is more conservative than either Hughes or Mathias on such issues as national defense, relations with the Soviet Union and immigration, organizers said.

A liberal-conservative hybrid, they added, would appeal to Maryland Democrats who crossed party lines to vote for President Reagan in last November's election.

"In a three-way race, we can win," said Terrence McGuire, a Seat Pleasant physician and Democrat who is one of the group's organizers.

Naturally enough, leaders of the state's two major political parties were more circumspect about prospects for a third-party candidate.

"It would not have an effect on either Mathias or Hughes, two of the most popular politicians in this state," said Maryland Republican Chairman Allan C. Levey. "Sam Bogley is identified with single-issue politics -- abortion -- but Maryland is not a single-issue state. Many of the antiabortion people would vote for Mac or Harry Hughes anyway, unless it's just a protest vote."

State Democratic party chief Howard J. Thomas said, "If a conservative Democrat ran for the U.S. Senate or any other office, I would hope he would do so within the party, through the primary election."

Conservative Democrats say avoiding the party's primary in the summer of 1986 would be a smart move, given the strength a politician such as Hughes would have in that Democratic party contest.

But mounting an independent challenge could be even more difficult, they said. The candidate would need more than 60,000 verified voter signatures to win a spot on the general election ballot, and tens of thousands of dollars for publicity.