House Speaker Thomas P. (Tip) O'Neill Jr. announced last night that he will ask the House to declare vacant the seat held by ailing Rep. Gladys N. Spellman (D-Md.).

O'Neill acted after receiving a report yesterday from the Capitol physician, Dr. Freeman H. Cary, in which Cary concluded that Spellman "is unable to resume service for this term of Congress." Cary, who went to Walter Reed Army Hospital to see Spellman on Thursday, said last night that it is "unlikely" that Spellman will ever be able to serve in Congress again.

Spellman was overwhelmingly elected to a fourth term on Nov. 4, less than a week after she was stricken with massive heart arrest. The speaker, who had said he wanted to give Spellman all the time she needed to recover from the Oct. 31 attack that left her semiconscious, said the congresswoman's husband, Reuben, told him on Thursday that "based on the most recent medical evaluation available, Mrs. Spellman's long-range prognosis for recovery appeared negative." That night Reuben Spellman told a gathering of his wife's supporters that there is a strong possibility he will seek to succeed his wife, but he is expected to face stiff competition both within the Democratic Party and from Republicans.

An aide to O'Neill said the Democratic leadership of the House will offer a resolution early next week, perhaps Tuesday, to declare the seat vacant. The action would be a rare one, usually invoked only upon the death of a member.

In Annapolis, state election board officials said Gov. Harry R. Hughes must issue a proclamation setting up a special election within 10 days of being notified of the vacancy. A primary must be held on a Tuesday no sooner than 35 days after the governor's order, with a general election to follow no sooner than another 35 days. If Hughes acted next Wednesday, a primary could be held as early as April 7 and a general election in mid-May.

After Spellman's staff got the news yesterday afternoon, in Room 308 of the Cannon House Office Building, a weary Edna McLellan, Spellman's top aide and alter ego, said, "it certainly isn't the happiest of times. We all adore Gladys, but these are the facts."

McClellan added that Reuben Spellman had known by looking at his wife that she was going to be unable to return. "But he isn't a doctor. We needed confirmation from a physician, and that's what this is."

Although the action by O'Neill was not unexpected, the speed with which he acted cought Spellman's staff and supporters by surprise.

"We knew it was coming after Mr. Spellman talked to the speaker on Thursday," said press aide Art Yeager, "but we didn't know it would be this soon."

Cary consulted with Spellman's physician and observed her at Walter Reed Army Medical Center, to which she was transferred last month from Prince George's County General Hospital.

The 62-year-old congresswoman, who had shown few signs of ill health before being stricken while attending a Halloween party in her home town of Laurel, has been in a sleep-like state since then, unable to move, speak, eat, talk or in any way communicate or indicate that she is aware of her surroundings.

Yesterday's announcment by O'Neill is likely to set in motion some intensive jockeying, both behind the scenes and in public, for the 5th District congressional seat, which the popular Spellman could probably have kept as long as she wanted.

In winning reelection to her fourth term in November, Spellman got more than 80 percent of the vote against a Republican novice. Seasoned politicians had dared not challenge Spellman, a consummate politician who began her career as a PTA volunteer and became one of the most effective campaigners in Maryland politics.

Whether her popularity would be transferrable to her husband is unknown. Asked about Reuban Spellman's prospects yesterday, Maryland Democrat Party Chairman Rosalie S. Abrams said: "Why not? It's been done for a number of congressmen whose wives have run. He would certainly be well qualified, but I don't think others would step aside for him. He would certainly be opposed."

One Democratic candidate will be State Sen. Edward T. Conroy, who was the party's unsuccessful nominee for the U.S. Senate last fall. "I'm a candidate," he declared last night, adding: "I've never been one to hesitate. cClearly, I'm the best candidate."

Another Democratic contender might be former State Senate President Steny Hoyer, who has been practicing law since his defeat in the 1978 gubernatorial primary.

Allan Levey, the Maryland Republican chairman, who was roundly criticized earlier for suggesting that the GOP begin looking for someone to run for the seat, said last night, "I think it's a good move by Mr. O'Neill at this time in Congress when key issues are coming up and the district needs representation."

Levey said he expected there would be four leading contenders for the Republican nomination: John Burcham, former Prince George's County councilman who was beaten by Spellman in 1974 and 1976; Larry Hogan Jr., the 25-year-old son and aide of the Prince George's county executive who held the congressional seat before Spellman; Bowie Mayor Audrey Scott, and Kevin Igoe, the political unknown who was trounced by Spellman last November.

Burcham offered the only serious challenge to the popular Spellman in her four campaigns. In 1974, he polled 40,805 votes to her 45,211, but two years later he did poorly, winning 57,057 votes compared to her 77,836.

Prince George's County Executive Lawrence J. Hogan, whom Spellman succeeded after Hogan resigned the seat to run for governor, reacted to yesterday's announcement by O'Neill with bitterness, saying, "what they have been saying all this time, that's she getting better, that's really terrible to do that kind of thing and mislead the public."

Parris Glendening, Prince George's County Council chairman, predicted that "you're going to have a scramble for the seat. I think you'll find half a dozen major contenders in the Democratic primary."

The name of Reuben Spellman was mentioned by 5th District Democrats shortly after his wife was stricken, but in those early days, he was viewed as a "caretaker" candidate who could win the seat until she recovered sufficiently to run on her own again.

With the announcement by Dr. Cary that it is unlikely Mrs. Spellman ever will run again, a Reuban Spellman candidacy might be viewed skeptically by party pros, who fear an all-out assault by the Republicans.

State Republican leaders say they have been promised "as much money as needed" by the Republican National Committee to try to convert the seat into the GOP column.

The House voted last month to continue paying Spellman's staff, and to pay her as a member-elect, even though she was never sworn in as a member of the 97th Congress.

Walter Reed spokesman Peter Esker said yesterday that Spellman's condition is "unchanged -- that is, in a semiconscious state."

Spellman's staff has worked on constituent problems throughout her illness, and several said yesterday they will continue to handle problems for district residents until her successor is chosen. But they had no comment on whether they would work in behalf of Reuben Spellman, a retired federal worker with no legislative experience.

McClellan, who has worked for the congresswoman since she was a member of the Prince George's County Council, stood outside the congressional office last night and said that Spellman would be remembered for "her integrity, honesty and ability to get something done."

Inside, the telephones were ringing incessantly, but no one bothered to answer them.