Steny H. Hoyer was sworn in yesterday as congressman from Maryland's 5th Congressional District, and said he intends to be "a vigorous advocate" of federal workers, in the tradition of ailing former representative Gladys N. Spellman.

Hoyer, a 41-year-old Democrat, won the seat after it was declared vacant because of Spellman's incapacity by beating Republican Audrey Scott in a special election last month.

In his maiden, one-minute speech on the House floor just after taking the oath from House Speaker Thomas P. (Tip) O'Neill Jr., Hoyer was relaxed and confident. He acknowledged the presence of his relatives and friends in the gallery, which prompted O'Neill to bang the gavel and say, "the gentleman may not address people in the gallery."

Later, Hoyer joked that "I've got some rules to learn. This obviously isn't like the Maryland Senate [in which he served 12 years], where everyone introduces their friends from Catonsville."

In his floor speech, Hoyer admitted that he still lives in the adjoining 4th Congressional District of Republican Marjorie S. Holt. He said he had kidded Holt, who was sitting in the front row with other members of the Maryland delegation, that "if you don't vote the way I like on the House floor, I will vote against you [at the polls next year]. He said Holt responded by threatening to redistrict him out of her district. "That's fine with me," Hoyer said. The fact that Hoyer lives two blocks out of his own district was a minor issue in the campaign.

At a press conference in the Speaker's office following the brief noon ceremony, O'Neill said Hoyer will get his requested committee assignments, Post Office and Civil Service, on which Spellman was a subcommittee chairman, and Banking, Finance and Urban Affairs.

Hoyer said his joy was tempered because "Gladys Spellman is not here. I always supported her. She was one of the most able and effective members of Congress."

Spellman has been semiconscious since suffering heart arrest last Oct. 31, while she was campaigning for a fourth term. She was overwhelmingly reelected the next week, but on Feb. 24, the seat was declared vacant, the first time in history a vacancy had been declared in the House as the result of disability.

Hoyer said he has hired two key members of Spellman's staff, Edna McLellan as administrative assistant and Betty Richardson as senior case worker, "to insure the continuity of service Mrs. Spellman was rightfully known for."