Rep. John B. Anderson of Illinois, the liberal who had seemed mired in fourth place in the Republican presidential derby, tonight advanced into the front rank of the race by strong showings that apparently left him just short of victory in the Massachusetts and Vermont primaries.

Anderson led in both states throughout the counting tonight, but slipped a few hundred votes behind Ronald Reagan in Vermont after midnight and trailed George Bush by a similarly slim margin in Massachusetts. Reagan was the apparent winner in Vermont, but the final outcome in Massachusetts remained in doubt.

Sen. Howard H. Baker was a badly beaten fourth in both states, a showing that deepened the gloom in his already faltering campaign. The other Republican contenders, John B. Connally, Sen. Bob Dole and Rep. Philip M. Crane, trailed as they had earlier in Iowa and New Hamsphire.

Former president Gerald Ford -- whose possible entry into the race burst on the eve of the two primaries -- was the recipient of about 1 per cent of the Massachusetts votes, all on write-ins.

With 86 percent of the precincts reporting in Massachusetts, Bush and Anderson had 31 percent each, Reagan 29 percent and Baker 5 percent. In Vermont, with 99 percent of the precincts counted, Reagan had 31 percent, Anderson 30 percent, Bush 23 percent and Baker 13 percent.

Indications were that more than 1.1 million Massachusetts voters went to the polls today, substantially more than the 923,000 who voted in 1976. The increase was proportionally greater in the Republican primary, apparently indicating the flow of independents in that direction.

Anderson told a jubilant gathering of his young supporters that his "campaign of ideas" had caught fire here and the response in these states would give him the impetus to continue in the Midwest and West.

But Bush told reporters in Charleston, S.C., that the results meant he had "stopped the hemorrhaging" from his loss to Reagan in New Hampshire, and had once again made it 'a two-man race" between himself and Reagan.

"I give John Anderson credit," Bush said. "He beat me, but he had a crossover vote that he won't have in other states, and he's not really running a national campaign."

In Los Angeles, Reagan said today's primaries were "not my territory, so a three-way tie is great." Like Bush, he said Anderson had "aimed well at the liberal Democratic crossover vote. . . . He will not get those votes elsewhere."

Whatever the final result, Anderson's showing gave another dramatic and unexpected turn to the exciting 1980 GOP race.

The congressman presented himself as the "different" Republican contender, criticizing the others for backing sharply increased defense spending and renewed draft registration. Anderson proposed, instead, that the United States reduces its need for Middle Eastern oil by imposing a 50-cent-a-gallon tax on gasoline and using the proceeds to cut Social Security taxes by 50 percent.

For Anderson, the results tonight were a vindication of his belief that he could form what he called "a new coalition" of Republicans, independents and -- in Vermont -- Democrats as well, behind his candidacy.

An NBC News poll of voters leaving the precincts today indicated that self-identified independents comprised about half the voters in the Republican primary in Massachusetts. Those independents gave 46 percent of their votes for Anderson, 26 percent for Bush and 19 percent for Reagan. The independents were younger than the registered Republicans.

The same survey showed Anderson got 60 percent of the liberals, and split the moderte vote with Bush, with Reagan third.

Surprisingly, Reagan got only 38 percent of the votes of self-identified conservatives, while Bush took 34 percent and Anderson 17 percent.

Those who decided late in the process -- after New Hampshire -- went disproportionately for Anderson, giving him 44 percent of their votes.

Anderson rolled up big margins over Bush and Reagan among the Republicans in Boston, Brookline, Watertown and other suburbs. Bush made his strongest showing in Springfield, Pittsfield and other western Massachusetts towns.

Tonight's showing is certain to boost the flow of contributions to Anderson and bring him more volunteers as well. Earlier today, John Topping, an Anderson campaign aide in Washington, said the Illinois congressman was receiving $25,000 to $40,000 a day and that samplings of direct-mail lists were encouraging enough -- with a 4 percent return averaging $25 a person -- to allow them to plan to drop three million appeals into the mail in coming weeks.

Anderson himself angrily rejected Bush's description of him as a "one-state" candidate, saying he will be on the ballot in at least 17 states.

However, Anderson is skipping some traditionally conservative states, and has failed to file in some of the states where progressive Republicans normally find delegates.

Anderson has made no effort in the four southern states -- South Carolina, Florida, Georgia, and Alabama -- that vote in the next seven days, although his campaign manager said tonight he may go into Florida.

He filed no delegates of his own for the New York primary on March 25, and did not come up with enough signatures to qualify for the beauty contest in Pennsylvania April 22.

The Anderson scenario calls for a strong showing in his home state of Illinois on March 18, in Connecticut on March 25, in Wisconsin on April 1, and then in the last spring primaries in Indiana, Michigan, Ohio, New Jersey and California.

Several of these states have "open primaries" where persons other than registered Republicans may vote in the GOP primary. That is the case in Illinois, Wisconsin and Michigan.

In California, Topping said, Anderson hopes to run a major registration drive aimed at enrolling 500,000 to 800,000 young people and independents as Republican voters before the deadline on May 4 passes in Reagan's home state.

Reagan was not hurt badly by today's results, according to his managers and most neutral observers. By finishing among the leaders in two New England states known for their tradition of liberalism, the longtime conservative standard bearer showed there is a degree of support for his views even in alien territory.

Reagan expects to clean up on the opposition in the four southern states and carry the momentum of those victories into what now looks like a true Republican showdown in Illinois.

Bush came away from the day's voting without the pair of victories he had hoped for -- before his drubbing by Reagan in New Hamsphire -- but he found some consolidation in the returns.

At this point, Bush most needed something to slow down Reagan, and the fact that it was Anderson who did it did not disturb Bush's managers.

"The guy we've got to beat is Ronald Reagan," Bush compaign manager James A. Baker III told reporters here. David Kenne, Bush's political director, said he thought the Anderson candidacy would stumble in less liberal GOP constituencies and that eventually it would be a Bush-Reagan race.

But Keene conceded that Bush had hoped to make it such a contest before April 1, and said today's results showed that "Anderson will be there" to divide the moderate vote with Bush in both Illinois and Wisconsin, increasing Reagan's chances for a plurality victory.

The candidate hurt most by today's vote was Baker, who announced in Washington tonight that he was cutting back and realigning his campaign staff.

Despite the help of Vermont Gov. Richard Snelling, Baker was unable to get better than a fourth place finsih in that state. And his indicated 5 percent of the vote in Massachusetts makes it even harder for him to raise funds for Illinois and later primaries.

"Baker has more staying power than anyone else," said Douglas Bailey, his media consultant. "But if we can't get votes, we can't raise money either."

Bush was the first candidate to organize in New England, exploiting family, college and party contacts stemming from his Milton, Mass., birthplace, his days at Andover and Yale and his chairmanship of Republican National Committee. Early polls showed him leading the field in both Massachusetts and Vermont and enjoying the advantage of superior organization as well.

But Anderson began working the New England college circuit early in his campaign and fashioning an appeal -- particularly in Massachusetts -- aimed at the large number of independent voters who can cross into either party's primary.

Reagan began with a conservative base of support, estimated at slightly more than one-third of the normal Republican primary vote in both states, and capitalized on the momentum of his New Hamsphire victory. Although he spent far less time in Massachusetts and Vermont than did either Bush or Anderson, Reagan campaigned in the two states vigorously on Sunday and Monday to exploit his New Hampshire win.

Massachusetts sends 42 delegates to the national convention, apportioned on the basis of today's vote. Vermont sends 19 delegates who will be chosen at the May 24 state convention. Ten of those delegates will be pledged for the first two convention ballots to the winner of today's primary, if that man has at least 40 percent of the votes in the official canvass.

Today's voting winds up the first phase of the New England contest.Maine is still in the first stage of choosing its state convention delegates who, in turn, will elect national convention delegates. Bush is leading in the results there thus far.

Connecticut holds its presidential primary on March 25, and Rhode Island, the last of the New England states, holds its primary, along with several other states, on June 3.