Walter Wellesley (Red) Smith, the Pulitzer Prize-winning sports columnist of The New York Times, died yesterday at the age of 76 at Stamford Hospital in Stamford, Conn., after a brief illness.

He had been in failing health in recent years after undergoing surgery for cancer of the colon. His death was attributed to a vascular event.

Mr. Smith, who resided in New Canaan, Conn., won the Pulitzer Prize for commentary in 1976. He was cited for the erudition and literary quality of his column, "Sports of the Times," which appeared four times a week and was syndicated to about 500 newspapers worldwide. He also wrote six books and numerous articles for magazines.

He was best known for his graceful, stylistic writing and the ease with which he could turn a phrase.

Mr. Smith joined The Times in 1971, after working for Publishers Newspaper Syndicate. He had become a nationally syndicated columnist for the New York Herald Tribune, where he worked from 1945 until that newspaper's folding in 1966.

The son of a wholesale grocer, Mr. Smith was born in Green Bay, Wis., Sept. 25, 1905. Mr. Smith graduated from Notre Dame University in 1927 and joined the Milwaukee Sentinel as a copy editor that year. He joined the copy desk of the St. Louis Star in 1928 and moved to sports when, as he was to recall, "one day they fired the sports department." In 1935 he joined the Philadelphia Record as a sports columnist.

In his final column for The Times, which appeared Monday, Mr. Smith wrote that he was reducing his column output from four a week to three. He reflected on the change--and his unique world--in that piece.

"The seven (columns) a week routine was in Philadelphia, which reminds me of the late heavyweight champion, Sonny Liston. Before his second bout with Muhammad Ali was run out of Boston, Liston trained in a motel in Dedham.

"I was chatting about old Philadelphia days with the trainer, Willie Reddish, remembered from his time as a heavyweight boxer in Philadelphia.

" 'Oh,' Willie said, apropos of some event in the past, 'were you there then? '

" 'Willie,' I said, 'I did 10 years hard in Philadelphia.'

"There had been no sign that Liston was listening, but at this he swung around. 'Hard? ' he said. 'No good time? '

"From that moment on, Sonny and I were buddies, though it wasn't easy accepting him as a sterling citizen of lofty moral standards."

Although Mr. Smith was best known and admired for his essay-style columns on everything from fishing to baseball, he took many strong stands. He was particularly sharp in his criticism of baseball's management and sided strongly with the players during last year's strike.

Mr. Smith was involved in a major controversy in January 1980 when editors of The New York Times killed a column he had written calling for a U.S. boycott of the Moscow Olympics. "Sentiment in favor of a boycott would spread as Soviet tanks and trucks press on with their bloody work in Afghanistan ," Smith had written. The previous week Mr. Smith had written a similar column calling for a boycott. The second column, Times editors felt, "sounded like a crusade."

Mr. Smith disagreed with the editors' decision, but called the flap "a bloody bore."

Favorite topics on which he wrote were baseball, horse racing, boxing and fishing. He rarely wrote on basketball or hockey.

When someone once told him how much he envied the ease with which Mr. Smith's words seemed to flow, he responded: "All I do is open a vein and it comes out one drop at a time."

Dave Anderson, who like Mr. Smith was a sports columnist for The Times and a Pulitzer Prize winner, said of his colleague: "He did what he always wanted to do and he did it better than anybody. He was the best sportswriter we ever had."

In his last column, Mr. Smith wrote that he had often been asked who was the best athlete he had ever seen, and the one he liked most. He responded: "Both questions are unanswerable, but on either count Bill Shoemaker, the jockey, would have to stand high . . .

"There were, of course, many others, not necessarily great. Indeed, there was a longish period when my rapport with some who were less than great made me nervous. Maybe I was stuck on bad ballplayers. I told myself not to worry.

"Some day there would be another Joe DiMaggio."

Mr. Smith is survived by his wife, Phyllis Warner Weiss Smith of New Canaan; a son, Terence, a reporter in the Washington bureau of The New York Times, and a daughter, Catherine Halloran of Graston, Wis.

A memorial service is scheduled for Wednesday at St. Patrick's Cathedral in New York City at 10:30 a.m.