Senate Democratic leaders moved yesterday to break a four-month Republican filibuster against constraints on the "Star Wars" strategic defense system that are included in the defense authorization bill for fiscal 1988, and Minority Leader Robert J. Dole (R-Kan.) conceded their effort may succeed.

If successful in a cloture vote Tuesday on the defense bill, Democrats could achieve a significant breakthrough in the legislative logjam that has gripped the Senate for most of this year.

But they face the prospect of a hard-to-override veto of the defense measure so long as it contains constraints on development of President Reagan's Strategic Defense Initiative.

They are also confronted with GOP objections blocking consideration of legislation to expand Medicare to cover costs of catastrophic illnesses, and they remain stymied over another major Democratic initiative aimed at limiting congressional campaign spending.

In a sixth cloture vote on the campaign spending measure, the Senate yesterday voted 53 to 42 to limit debate, picking up two votes but still falling seven short of the 60 necessary to cut off a Republican filibuster. The new additional votes for cloture came from Sens. Nancy Landon Kassebaum (R-Kan.) and Ernest F. Hollings (D-S.C.).

The spending measure is expected to be shelved for the year if, as expected, it fails to muster 60 votes in a seventh cloture attempt on Tuesday.

The outlook for cloture on

the defense bill is considerably better.

Democrats have 59 votes to break the filibuster on that bill and expect to pick up the support of Sen. Mark O. Hatfield (R-Ore.), an arms-control supporter who previously voted with most other Republicans to sustain the filibuster. Hatfield has declined to say how he will vote Tuesday although a spokesman said he is "inclined" to support cloture.

"I think the votes may be there on cloture," said Dole, who has orchestrated the GOP efforts to block several key Democratic initiatives, including arms constraints in the defense bill.

But Democrats would probably have no votes to spare, prompting a warning from Majority Leader Robert C. Byrd (D-W.Va.) that a "spotlight" will be on absentees on Tuesday if the cloture attempt fails. With Sens. Joseph R. Biden Jr. (Del.), Albert Gore Jr. (Tenn.) and Paul Simon (Ill.) engaged in campaigning for the Democratic presidential nomination, Byrd has had trouble turning out his full complement

of 54 Democratic votes on any

issue.

Several Republicans, including Sen. William S. Cohen (Maine), a senior member of the Armed Services Committee, said prospects for passage of the defense bill will be good if cloture is invoked Tuesday, although further delaying tactics are likely.

At issue is language included by Armed Services Committee Chairman Sam Nunn (D-Ga.) and Sen. Carl Levin (D-Mich.) to require congressional approval for reinterpretation of the 1972 Antiballistic Missile (ABM) Treaty to allow expanded testing and development of the space-based SDI.

The Republican filibuster has been aimed at forcing the Democrats to delete the provision, and the White House has threatened a veto if Congress' final version of the bill includes the language. The House has already approved it in slightly different form. In any veto confrontation, Reagan is expected to have an upper hand because it would be difficult for the Senate especially to produce a two-thirds majority to override a veto on the arms provision.

In preliminary debate on the measure yesterday, Nunn reiterated his intention to help block action on confirmation of Supreme Court nominee Robert H. Bork until the Senate disposes of the defense bill, and Byrd said the administration could speed action on Bork by helping the Senate clear its decks of other business, including the defense measure.