Nowhere in America is Edward M. Kennedy's speech Monday being awaited with more anxiety than in a ramshackled house on Columbia Street here, where a dozen staff members -- just removed from the payroll in an economy move -- are manning the front line in Kennedy's next battle for political survival.

The Maine town caucuses on Feb. 10 will provide the first direct referendum on this latest, and perhaps last, effort to revive the Massachusetts senator's failing challenge to President Carter's renomination.

The latest soundings in the state indicate the Kennedy has slipped behind Carter in a contest Kennedy himself has said he must win.

His backers here are counting on Monday's speech -- to be carried on the state's three television stations -- to spark a last-ditch drive to avert what now appears to be a likely and devastating defeat.

"There is still ample time for him to campaign here intensively," Gov. Joseph Brennan, Kennedy's chief backer, said Saturday, in dismissing the importance of the senator's cancellation of Friday's and Saturday's scheduled appearances in the state in order to prepare for his television talk. "And if he campaigns intensively here, he can win."

Another key Kennedy strategist cautioned, however, that "up until now, his candidacy has not articulated the presidency he has in mind, and he's got to do that in this speech, for us to have a chance."

Three uncommitted but knowledgable Maine Democratic leaders said in separate interviews this weekend that Carter has become "a clear favorite" in the caucuses.

That judgment is supported by the findings of an extensive and ongoing survey of Maine voters, being taken by a private organization not involved in the presidential race. Among the Democrats in the weekly samplings of about 500 Maine voters, Carter has moved from a 10-point deficit a month ago to a lead of about 10 points in the last two weeks' surveys.

There are the usual caveats about caucus results not necesarily reflecting broader opinion, but the trend is clear. As in Iowa, the survey officials report that Carter is being bolstered by a surge of approval for his leadership in the tense international situation while Kennedy is being hurt by an undercurrent of personal criticism, reflected in the comment that "he's not my kind of man."

Except in Portland, Brennan's home base, and in Biddeford, where local Democratic legislators are selling Kennedy to their blue-collar, Franco-phone constituents, the pollster said, "I simply can't find any hardcore Kennedy support."

In the past, the Maine caucuses have been of no great import, because they were scattered over several weeks (as they still are in the GOP this year) and they determined the allocation of relatively few national convention votes. Only 6,800 of the nearly 250,000 enrolled Democrats attended the 1976 caucuses.

But the decision of Maine Democrats to consolidate the caucuses on a single date this year, and the need for Kennedy to win over Carter to offset his shellacking in Iowa, have upped the ante in the Feb. 10 contest.

Most Democrats here expect the turnout a week from Sunday will be at least double that of 1976 and the stakes are infinitely larger.

Kennedy stopped in Portland on his first day as a candiidate. He visited Lewiston, Saco, Presque, Isle, Bangor and Sandford on another two-day swing, and is scheduled to return to Maine at least twice more before the caucuses. Brennan has put his own top political aide, Connie LaPointe, into the Kennedy campaign and former senator William D. Hathaway also is campaigning hard for his former colleague.

Hathaway is just one of many Maine Democrats for whom Kennedy has labored over the years. And the link is more than geographical. As Peter Meade, the young Boston politician who heads Kennedy's 15-member staff here, said:

"If there is a state anywhere where the Kennedy candidacy can be put together on the basis of his belief that government should help people, this is the state. It is one of the poorest in the country. It is sensitive to oil prices and oil profits. It really needs government help."

While Maine ranks 36th among the states in per-capita income, energy cost differentials make its real income lower.

On the other hand, there are many factors that make it less than hospitable for Kennedy. Its population is 79 percent rural and even Maine's Democratic Party is rather conservative, as witness the fact that Brennan was elected in 1978 on a platform of no tax increases.

"Maine doesn't have any of the traditional Kennedy constituencies," Meade acknowledged. "Labor is weak here. There are no big cities, no minority populations."

"There is a big Catholic population," Meade said, "but I'm not sure that works for us."

Abortion is an issue and so, many in both campaigns acknowledge, is Chappaquiddick.

Chip Bishop, a young Rhode Islander and 1976 Carter campaign veteran, whose French-Canadian and Yankee parentage makes him perfect for Maine, opened a headquaters last fall at a time when the Kennedy-Brennn partnership seemed invulnerable.

But, in much the same patter as Illinois, the Carter operatives have been able to enlist powerful support despite the opposition of the state's top elected Democrat.

Vice President Mondale signed up Maine's secretary of state, Rod Quinn, described by pro-Kennedy national committeewoman Nancy Chandler as "the best organizer in the state," to work for Carter. John Martin, speaker of the Maine House, came aboard, and the president, through phone calls, enlisted eight of the 13 Democratic state senators and 11 of the 16 county Democractic chairmen.

The president also has the tacit support of Sen. Edmund S. Muskie, who is publicly neutral, but whose Maine staff aides are helping the local Carter campaigns.

Muskie said here on Jan. 7: "I have served in Washington under six presidents, and I've never known another who's been so responsive, sympathetic and helpful in dealing with Maine problems as President Carter."

Since the Kennedy challenge became evident, the administration has reversed the decision to phase out Loring Air Force Base and has pumped in millions of dollars worth of housing, transportation and community development aid.

Now the state is being flooded with campaigners and staff workers. Since the middle of last week, Rosalynn Carter, Chip Carter and Joan Mondale have been campaigning for the president (along with Carter's "Aunt Sissy" Dolvin, who moved to Maine this year, just as she did before the 1976 caucuses), while Kennedy has had his son Teddy sister-in-law Ethel and niece Caroline up here.

With every day bringing additional recruits from Massachusetts for Kennedy and from Washington, D.C. for Carter, the only limit on the number of people working to increase the caucus turnouts is the fear in both camps that the independent Maine residents may resent too much pressure from outsiders.

"I tell people," said Rod Quinn, "that their one vote in the caucus in February is worth 1,000 votes in the election in November."

With Kennedy facing the possibility of a devasting defeat from the few thousand caucus-goers, Quinn's statement is no exaggeration.