Rep. Marjorie S. Holt (R-Md.), in an emotional speech on the House floor, announced yesterday that she will not run for reelection next year, a move that immediately set off a scramble for Maryland's 4th Congressional District seat.

"After some months of prayer and deliberation, I'm announcing today that I will not seek reelection . . . , " Holt, 64, told her colleagues. "I simply want more time with my wonderful husband Duncan and our seven grandchildren."

Holt, who has been one of the most conservative members of the Maryland delegation since her election in 1972, has been thinking of retirement for some time. She said in an interview yesterday that she was talked out of her retirement plans two years ago by the Republican leadership.

The White House, Republican leadership in Congress and in the state applied similar pressure this year, said Holt, but after much agonizing she decided to go ahead with her retirement plans.

Holt told her closest supporters Wednesday night at a small gathering at the Capitol Hill Club, and then made her decision public during a 30-second speech on the floor yesterday morning. Her voice broke with emotion, particularly as she spoke of the "many wonderful friends" she had made in the House. Her colleagues responded with a standing ovation.

Democrats greeted the news enthusiastically. Mark Johnson, a spokesman for the Democratic Congressional Campaign Committee, said the 4th District will be "one of our top opportunity races for 1986."

Republicans, meanwhile, said they were optimistic they could hold onto the seat, although they acknowledged that it would be tougher without Holt, who has been the district's only representative since it was created in 1972.

The leading Democratic contender appears to be Washington Bullets forward Tom McMillen, a former Rhodes scholar who has raised about $80,000. He said he will announce his candidacy on Sept. 10.

Holt and state Republican Party Chairman Allan C. Levey are backing Robert R. Neall, the minority leader of the House of Delegates. Neall said he will make an announcement in two weeks "and it isn't going to be any big surprise."

The 4th District race will be a good test of how the Reagan administration policies have been received by federal workers and by blacks, both present in large numbers in the district, according to Democrats.

The 4th District stretches from Anne Arundel County to southern Prince George's County, and includes Annapolis and several suburbs of Baltimore. Although Democrats exceed Republicans in voter registration, the district is a politically volatile area in which voters often cross party lines.

Holt has been a formidable vote-getter and has not received less than 60 percent of the vote since 1978. Democrats, however, said earlier this year that they had a shot at defeating Holt, because of McMillen's celebrity status.

Holt made made it clear yesterday she was not retiring because of concern about a challenge from McMillen, or because of ill health, another rumor that had been circulating.

"I feel great. I am not tired," she said. "I feel good enough to chop wood with President Reagan, and I can beat the tar out of Tom McMillen."

Most of Holt's colleagues said they were not surprised by the decision. "She mentioned it enough times," said Rep. Helen D. Bentley (R-Md.), who worked closely with Holt. "I'm unhappy about it."

"She's very active. She works very hard, and I think she just got burned out," said Rep. Roy P. Dyson (D-Md.).

Dyson also said that Holt took the job very seriously and thus took much of the criticism during campaigns very seriously. "After a while you say, 'Enough is enough,' " he said.

Holt, a lawyer, is described by colleagues as hardworking and tough-minded. When she was elected in 1972, she was the first Maryland woman to serve in Congress in 30 years.

She is the fourth ranking Republican on the House Armed Services Committee, where she has been a staunch supporter of defense spending increases. She also served on the Budget Committee for four years, where she advocated dramatic domestic spending cuts, and she has taken a leading role in several antibusing fights.

Now she says she wants to do some sailing and perhaps write her memoirs. She has been keeping a diary, and she jokes with her staff that they should each write a chapter and then turn it into a situation comedy.

Deciding to finally leave Congress "was a very difficult decision," said Holt. "But I think it's time. It's time to do something else.