Sen. Robert T. Stafford (R-Vt.) won his party's three-way primary Sept. 14, but his victory was not that impressive, and he has seen two key blocs of backers begin to slip.

Union members and environmentalists, many of them Democrats who crossed over to vote for Stafford in the primary, have begun to move toward support of Democratic candidate James Guest. This trend could place Stafford's seat in jeopardy.

The most serious blow came Sunday, when the Vermont Labor Council, representing 20,000 AFL-CIO members in the state, unanimously endorsed Guest.

The full impact of this is not yet clear, but 18 labor political action committees supported Stafford in the primary, and some of those will no longer do so, given the Guest endorsement.

Stafford, the senior senator, has been a down-the-line labor supporter for more than a decade. In the current Congress he consistently has helped block anti-labor initiatives in the Senate Labor and Human Resources Committee, so the lack of labor enthusiasm for his candidacy was striking.

"I don't think that seniority is that important," said one pro-Guest delegate. "If we can get Guest elected this year he will get seniority soon enough, and we'll have him there 20 or 30 years."

This delegate and several others noted that Stafford's pro-labor rating this year has been just 25 percent and that Stafford has been supporting President Reagan's economic program. The situation is not so clear with environmentalists, a powerful political voice in Vermont, but there have been indications that Guest is making inroads there also.

Shortly after the primary, for example, Carl H. Reidel, a University of Vermont professor and the chairman of the Vermont Natural Resources Council (VNRC), wrote to the League of Conservation Voters sharply criticizing national and local environmental groups for backing Stafford without considering Guest's qualifications.

Reidel attacked Stafford's environmental credentials, noting that he had voted to confirm Interior Secretary James G. Watt and Environmental Protection Agency administrator Anne M. Gorsuch and that he has failed to call for their resignations.

Reidel's is not the only position among Vermont environmentalists. In fact, they are split by the election.

Donald Hooper, executive director of the VNRC, said, "Fully 50 percent of those I talked to are fascinated with this nice dilemma and have not made up their minds."

This also has created a problem for the national environmental organizations. Thus the League of Conservation Voters will continue to support Stafford fully on the grounds, as a field worker put it recently, that Stafford's reelection is critical.

She noted that Brock Evans, president of the Audubon Society, recently called Stafford a "national treasure," and added that while Guest is attractive he does not have Stafford's record and, "more important, he could not replace Stafford as chairman of the Senate Public Works and Environment Committee."

Rey Post, Stafford's campaign manager, said the union endorsement of Guest will aid Stafford in his own party. "Republicans will realize that Stafford has to be their guy," he said, and he contended that Stafford will be able to retain the loyalty of the environmental movement.

This could turn out to be true. And Guest apparently still trails the incumbent. But the last two weeks have not been good ones for Stafford, and unless the next five are better he could lose his seat.