Stock prices were mixed yesterday in a session marked by pressure on two big-name glamor issues - International Business Machines and Digital Equipment.

The Dow Jones average of 30 industrials eased 3.89 points to 766.29 after a 3.47-point decline Tuesday. But gainers managed to outnumber losers by about a 7-6 margin on the New York Stock Exchange.

IBM fell 3 5/8 to 236 1/4. At mid-day the computer giant reported that its first-quarter earnings were up just 2.7 percent from the like period last year.

Digital Equipment, meanwhile, dropped 1 to 38 7/8 in active trading on a U.S. Patent and Trademark Office ruling that the company failed to disclose some pertinent information when it obtained a patent for computer tape in 1964.

Given IBM's status as the single most prominent holding of investing institutions, disappointment over the company's earnings tended to weigh down the market generally, analysts said.

In addition, they said traders continued to register a subdued responsed to President Carter's economic message.

In his address to newspaper publishers and editors on Tuesday, Carter outlined an anti-inflation strategy that put heavy emphasis on voluntary wage and price restraint by business and labor unions.

Analysts said investors seemed pleased that the president had once again ruled out mandatory wage and price controls.

Looming ahead for the market was the weekly report on the money supply due this afternoon from the Federal Reserve. Some analysts have forecast a big increase in the basic measure of the money supply that might encourage some credit-tightening moves by the Fed.

Yesterday's volume on the Big Board reached 26.21 million shares, up from 24.30 million in the previous session.

Nationwide turnover in NYSE-listed issues, including trades in those stocks on regional exchanges and in the over-the-counter market, totaled 30.04 million shares.

The NYSE's composite common-stock index slipped 0.04 point to 50.43.

Standard & Poor's index of 400 industrials gave up 0.19 point to 99.08, and S&P's 500-stock composite index was down 0.14 point to 90.11.

Cutler-Hammer, embroiled in a three-way legal battle following recent purchases of the company's stock by both Tyco Laboratories and Koppers Co., led the active list and climbed 3 3/4 to 44 3/4.

At the American Stock Exchange, the market value index set its seventh consecutive high since 1973 with a 0.19-point advance to 132.54.

The NASDAQ composite index for the over-the-counter market also set a 4 1/2-year high, rising 0.43 point to 108.87.