NEW YORK, JAN. 13 -- The stock market settled for a mixed showing today at the close of an erratic, sluggish session.

The Dow Jones average of 30 industrials, off more than 20 points in early trading and up more than 20 at its afternoon peak, finished with a net loss of 3.82 at 1924.73.

Advancing issues outnumbered declines by about 8 to 7 among New York Stock Exchange issues. Volume on the Big Board came to 154.02 million shares, down from 165.73 million in the previous session.

Meanwhile, the dollar closed mixed against key foreign currencies after gaining in the afternoon on a joint endorsement of currency stability by President Reagan and Japanese Prime Minister Noboru Takeshita. Gold fell and silver rose.

Noting the slow pace of stock trading, brokers said market participants were reluctant to take big positions in either the dollar or stocks in advance of Friday's scheduled report by the Commerce Department on the U.S. trade deficit for November.

Hopes for an improvement in the figures have been mingled of late with rumors that another disappointing report is in store.

Some Wall Streeters say investors have also grown increasing leery of unpredictable swings in stock prices linked to so-called index arbitrage strategies used by program traders.

Friday is a monthly expiration date for some options and futures that are part of these transactions.

Texaco fell 1 1/2 to 37 1/2. The stock stood at 40 1/2 when trading in it was halted on the NYSE for an announcement by the company that it may face claims for back taxes of as much as $6.5 billion.

Pennzoil, which has agreed to a proposed settlement with Texaco of the two companies' long legal battle, dropped 1 3/4 to 66 3/4.

Trans World Airlines lost 1 to 19 1/2. Interests controlled by Carl C. Icahn, TWA's chairman, have a large position in Texaco shares.

Digital Equipment dropped 2 to 122 3/4 on top of a 7 7/8-point tumble Tuesday, in selling attributed to uneasiness over the company's impending quarterly earnings report.

After the close in New York, the company reported fourth-quarter earnings of $2.48 a share, up from $2.02 in the comparable period last year but toward the low end of advance estimates on Wall Street.

As measured by Wilshire Associates' index of more than 5,000 actively traded stocks, the market increased $3.73 billion, or 0.15 percent, in value.

The NYSE's composite index of all its listed common stocks edged up .21 to 137.95.

Nationwide turnover in NYSE-listed issues, including trades in those stocks on regional exchanges and in the over-the-counter market, totaled 175.64 million shares.

Standard & Poor's index of 400 industrials lost 0.08 to 283.15, while S&P's 500-stock composite index rose 0.39 to 245.81.

The Nasdaq composite index for the over-the-counter market gained 0.71 to 332.68. At the American Stock Exchange, the market value index closed at 263.27, up 0.51.

In New York, the dollar ended at 126.82 yen, down from 127.45 yen Tuesday. Earlier in Tokyo, it finished at 126.30 yen, down from 128 yen Tuesday.

The dollar closed at 1.6370 West German marks in New York, up from 1.6340 marks Tuesday. Earlier in Frankfurt, it dropped to 1.6335 marks from 1.6369 marks Tuesday.

In London, gold ended at $482.50 an ounce, up from $481.75 Tuesday.

Silver increased to $6.79 an ounce from $6.76 on the cash market.