The Interstate Commerce Commission, which has a century-long history of approving railroad mergers, denied one of the biggest yesterday when it said no to a marriage of the Santa Fe and the Southern Pacific.

The 4-to-1 vote could signal a significant shift toward more regulation by a government agency that coal interests have labeled "the wholly owned subsidiary of the Association of American Railroads" and that the Reagan administration has proposed eliminating as unnecessary.

Only six years after deregulating the railroads, Congress is considering some re-regulatory legislation and the administration-backed Norfolk Southern purchase of federally owned Conrail has foundered in the House, largely on questions about its potential anticompetitive impact.

Only ICC Chairman Heather Gradison voted for the Santa Fe-Southern Pacific merger. Two of the commissioners opposing it cited the loss of competition that could result from merging two networks that run parallel through California's Central Valley and across the Southwest from Los Angeles to the Gulf of Mexico. The other two opponents did not explain their vote.

Approval would have produced a 25,000-mile rail system -- the nation's longest -- with combined revenues last year of $4.6 billion, second among railroads only to Richmond-based CSX Corp.

The ICC staff had recommended approval of the merger, but called for special conditions to ease concerns about competition.

The last time the ICC turned down a major railroad merger was in 1968, when it denied a proposal from the Chicago, Burlington & Quincy, the Great Northern and the Northern Pacific. However, that proposal was revised somewhat, resubmitted in 1970 and approved. The result is the Burlington Northern.

One of the spurs to railroad deregulation and legislative requirements for speedy ICC review of merger cases was the 11-year debate the ICC had in the 1960s and 1970s over the proposed merger of the Union Pacific and Rock Island. By the time it approved the merger, the Rock Island was so close to death that the Union Pacific declined to consummate the deal.

Andras Petery, a rail analyst with Morgan Stanley & Co., said the ICC "has responded to the backlash of deregulation and this case was no doubt a victim to that policy."

Frank N. Wilner, an economist for the Association of American Railroads, predicted that "the next Congress will be looking at securities, airlines, banking, trucking -- all these sectors that have had real or perceived difficulties -- and the commission, being a political animal, is responding to the signals it hears."

On the other hand, nobody thinks the decision is a good one for the future of Southern Pacific or its employes and shippers. "The Southern Pacific is a bankruptcy candidate," Petery said, "and the market solution will be rationalization with bankruptcy a possibility. How fast is the only question."

The two railroads' holding companies merged in 1983 to become the Santa Fe Southern Pacific Corp., but the Southern Pacific railroad was operated by an independent trust pending the ICC decision. Unless a court appeal overturns the ICC decision, the parent corporation must sell the Southern Pacific.

Santa Fe Southern Pacific attorneys argued before the ICC that the merger was essential for the Southern Pacific's survival. After the vote, Sante Fe Southern Pacific stock dropped by $2.25 a share to $28 a share. Standard & Poor's Corp. placed a $806 million in Santa Fe Southern Pacific bonds and other debt securities on "credit watch with negative implications."

John J. Schmidt, chairman and chief executive officer of the SFSP, told reporters, "I think the commission made a horrible mistake." But he said the decision is "not doomsday for Santa Fe Southern Pacific. There are many things we can do with parts or all of the railroads." He said a sale of Southern Pacific "could be" in parts, meaning the railroad as such might disappear but some of its routes would survive as parts of other lines.

A court appeal of the ICC decision would be tough to win, he said.

The vote came in a swift, 18-minute meeting that the reporters, lawyers and lobbyists who crammed the main ICC hearing room had expected to last for hours. Most analysts were predicting an approval of the merger with agonizing discussions over permitting other western railroads opposing the merger to use some of the tracks of Sante Fe-Southern Pacific.

That scenario was put to rest when Commissioner Malcolm M. B. Sterrett said that the public benefits of the merger "are far outweighed by anticompetitive" concerns and that the "conditions to ameliorate are far too clumsy."

Commissioner J. J. Simmons said simply, "The anticompetitive effects outweigh the public benefits." Commissioners Frederic N. Andre and Paul Lamboley remained silent. Gradison praised the proposed merger and called for the vote.

Gradison said later that "the action denying the merger represents an opportunity missed. If this commission is too timid to permit a combination which would result in some rationalization of the nation's rail operations, it should have at least recognized the deterrent effect of its actions on railroad initiative."

Andre's vote was, at first glance, the most surprising, because he is an unabashed free marketeer and once said in a commission meeting on trucking issues: "Bribes among principals are probably one the clearest instances of the free market at work. . . . A bribe . . . is an attempt to get around the rigidities imposed on the market by a government cartel."

He was not available yesterday to explain his vote, but sources familiar with his thinking said he found overwhelmingly offensive the conditions that would have been imposed to solve anticompetitive problems. The pure free-market solution would be to let a bankruptcy occur, the sources said.