A Maryland couple who tried to defeat the federal tax system's "marriage penalty" by getting quickie Caribean divorces were informed by the U.S. Tax Court yesterday that their scheme didn't work.

In a 22-page opinion, the tax court said that such foreign divorces would not be valid in the eyes of the Maryland courts. The results, the court said, is that despite two divorces, David and Angela Boyter of Elliott City remained husband and wife in 1975 and 1976, and therefore owe the Internal Revenue Service the extra $3,135.34 in income taxes they had hoped to avoid by claiming single status.

the Boyters, who married for the first time in 1966, promptly remarried after their first two divorces. They divorced for a third time in 1977, and have vowed to stay unmarried (although they live together) while crusading to reform the tax system.

"We will stay single until the law is changed," Angela Boyter said in a telephone interview last night.

When they were married, the Boyters found that the tax laws force married working couples earning comparable salaries to pay higher taxes than men and women who live together but remain single.

The tax court noted in its opinion that "after calculating how much matrimonial bliss was costing them, the couple traveled to Hati in 1975 and to the Dominican Republic in 1976 to obtain "perfunctory year-end divorces" before Dec. 31 of each year.

"Not surprisingly," the court said, couples who discover the so-called marriage penalty in the tax law find it both "onerous and unfair."

Despite their divorces, one of which was granted on the grounds of "incompatibility of temperments making life together unbearable," the Boyters never separated from each other and were always residents of the same house they had purchased together in 1967, Judge Richard C. Wilbur wrote in an opinion for the court.

Their sole intent -- which the IRS called a sham -- was to divorce on paper so they could save money as single taxpayers, the court noted.

The problem with their foreign divorces, the court said, was that the Boyters always resided in Maryland. And although the Maryland courts have never directly confronted the question, the tax court said it was convinced that either husband or wife would have to establish residence in the foreign country before such a divorce would be valid. in Maryland.

The tax court noted that six states and Great Britain have similar rules for foreign divorces.

"We are therefore convinced that despite the participation in the divorce by both the Boyters, the courts of Maryland would not recognize it as valid to terminate the marriage," Judge Wilbur wrote.

Angela Boyter was not persuaded. "I think we are probably going to appeal," she said in an interview.

"I don't want to be single . . . I think people have a right to marry if they want to" and not be subject to extra taxes, said Boyter, who described the marriage penalty as "bad social policy."

Boyter said she felt the court decision made it "even more imperative" that Congress act to do away with the marriage penalty.

Sen. Charles McC. Mathias Jr. (R-Md) and Rep. Millicent Fenwick (R-N.J.) have introduced bills that would accomplish that purpose.