The stock market ended its five-day rally yesterday as the Dow Jones industrial average dropped more than 50 points in heavy, seesaw trading.

The market opened lower and fell more than 100 points before staging a brisk rally in the last half hour to close at 1963.53, down 50.56. Volume was 228 million shares, compared with 176 million Monday.

Analysts cited profit-taking -- the Dow had recovered 220 points in the previous five days -- and investor pessimism about whether talks on reducing the federal budget deficit would bear fruit. "What this tells you is that the market is still nervous," said Michael Sherman, chairman of investment policy at Shearson Lehman Bros.

President Reagan said the stock market decline that began last month "is one warning that we can't afford to ignore," the Dow Jones news service reported. His remarks came during a swearing-in ceremony for the new Labor secretary, Ann D. McLaughlin.

"The stock market has been telling us that unless we pay attention to some unfinished business, it won't last," Reagan said, apparently referring to the nation's economic expansion.

Erratic movement was seen in other key U.S. financial markets. The dollar recovered a bit after slipping to record lows against the yen and mark. Bonds opened lower, then regained a bit of strength during the day.

Meanwhile, the New York Stock Exchange announced it will lift today some of the restrictions it imposed last month on program trading, a computer-based buying and selling strategy that many analysts have blamed in part for the speed of the 508-point plunge of Oct. 19.

The NYSE left in place other limits that could slow down any impact of program trading on the market. "Markets are now beginning to stabilize," the exchange said in a statement. "But because trading activity remains well above previous trading levels, it is prudent to return to normal trading procedures in stages."

Standard & Poor's 500-stock index fell 4.92 yesterday to 250.82. The American Stock Exchange composite index fell 6.24 to 255.49, with the average price per share down 28 cents. The Nasdaq composite index, which measures activity on the over-the-counter market, declined 7.66 to 320.67.

The falls echoed declines in the European markets overnight that saw London's key Financial Times index of 100 stocks lose 4 percent. The Tokyo market was closed due to a Japanese national holiday.

Some analysts attributed yesterday's drops to reports that negotiations between Congress and the White House over ways to reduce the federal budget deficit have stalled. Action on the deficit is widely considered key to solving the country's long-term economic difficulties.

Reagan said that everything except Social Security is negotiable in the budget talks. "One way or another, we're going to put the public menace of deficit spending behind bars," he said.

Many investors seemed to feel that nothing significant was happening, some analysts said. "They said that the president wasn't taking it seriously enough, doing enough," said Ira Penziner, account executive at Dean Witter Reynolds Inc. "Throughout the day we heard that."

Other analysts saw the hand yesterday of routine profit-taking after a rally. "It's only natural for some of those gains made on a cumulative basis over those five days to be retrenched," said Otto Grote, president of Derby Securities Inc. "That's exactly the thing one would expect."

On Oct. 20, the New York exchange asked members to suspend trading using the floor's automated order delivery system. It did not ban the practice if the firms' computer-devised orders were processed manually on the exchange floor, as a number of firms had the personnel to do.

On Oct. 22, the exchange broadened the request, asking that firms not carry out any type of program trading for their own accounts, though they could on behalf of others. Yesterday's move formally gives firms permission to resume the trading for their own accounts. It keeps in place the prohibition against using the automated delivery system.

Before the Oct. 19 plunge, program trading accounted for about 20 percent of the transactions on the exchange floor. It was not clear how many of them used the automated delivery system, however, and what volume of trading it might generate.