When Maine Democratic Chairman Harold Pachios went to lunch recently at the Portland restaurant owned by Tony DiMillo, the two men talked about the Feb. 10 Democratic town caucuses, where DiMillo was one of those who gave President Carter his victory over Sen. Edward M. Kennedy.

"I got to tell you," DiMillo told Pachios, "that for the first time in my life, I'll probably vote Republican in November. I'm worried about this economy but Teddy was just too liberal for me."

That conversation highlights an easily overlooked political truth that is well understood by leaders of the Carter campaign and other Democrats: despite the drubbing Carter has given Kennedy and his other intra-party challenger, Gov. Edmund G. (Jerry) Brown Jr., the president may be vulnerable to upset in the general election.

That is not the surface impression in a week that will see Carter come off a three-state primary sweep in the South, and a series of expected caucus victories from Delaware to Hawaii, into a probable win next Tuesday in the vital Illinois primary.

But top Democrats point to a number of warning signs that suggest, in the words of Carter's pollster, Patrick Caddell, the president may face "a very difficult general election situation."

Here are some of the worrisome factors:

Despite the tendency to rally around a president in a time of international tension, Carter's ratings on handling the economy, working with Congress and getting things done are still scrapping the dangerous lows of last autumn, and his overall job rating has begun to decline.

There is evidence that the runaway inflation that has forced Carter to revise his economic game plan is finally beginning to emerge as a voting issue. At the same time, Carter's efforts to free the hostages in Iran, force a Soviet withdrawal from Afghanistan and promote peace in the Middle East -- all sources of strength the past six months -- have produced few results and could blow up in his face. t

Vital Democratic voting blocs -- liberals, young people, Jews -- show a readiness to defect to some of Carter's possible Republican opponents.

The intra-party wounds of the Carter-Kennedy struggle seem lasting in several of the states where the war has been fought, raising fears of further Democratic defections in November. "The longer this goes on," Caddell said, "the more damaging to Carter."

The possibility of a significant third-party effort, perhaps led by liberal Republican John B. Anderson, could give dissident Democrats an outlet for hurting Carter in the fall.

And, finally, Carter's long absence from the campaign trail has left some of his advisers worried that he may be an awkward, out-of-shape fighter when he finally begins his personal drive for a second term.

All of these fears may be exaggerated.Although Carter's most recent Gallup poll approval rating, 52 percent, is down 9 points from the Iranian crisis high of December, and down 6 points in just a month, it is still an exact match for Richard Nixon's standing in February 1972.

Nixon went on to win a 49-state landslide over George McGovern that November, and most Democrats interviewed in recent days appear confident that Carter would be a solid favorite in a two-man race against the current Republican front-runner, Ronald Reagan.

"If they nominate Reagan," said Democratic National Chairman John C. White, "I think you can project a lot of Carter's current strength to a general election. The perception of Reagan as a plausible alternative president just hasn't permeated to the country."

A Gallup poll completed early this month showed Carter leading Reagan, 57 to 34 percent, and beating second-place Republican George Bush, 57 to 32 percent.

But when pollster Lou Harris reported on Sunday that former president Gerald R. Ford led Carter in a trial heat, 54 to 44 percent, there was no great skepticism from anyone, even though it was the first time in six months that any Republican finished in front of the president.

"The basic fact," said Republican National Chairman Bill Brock, "is that Jimmy Carter is extremely vulnerable."

Almost all analysts of the shifting tides of public opinion comment on the fact that, despite Carter's near sweep of the primaries and caucuses, in critical areas of job rating, he remains dangerously low.

"The paradox," said Peter D. Hart, the pollster for the Kennedy canpaign, "is that every Tuesday seems to buttress his [Carter's] position, but the voters' attitudes are not very good. The structure of his support looks sound on the outside, but inside, the termites have really been at work."

In New Hampshire, for example, where Carter dealt Kennedy a crippling blow, a CBS-New York Times poll of voters found 72 percent disapproval of Carter's handling of the economy.

He has been consistently rated low as well for his dealings with Congress and his overall ability to get things done.

Caddell said Carter has been immunized against the potential damage of the economic issue because "people don't believe Kennedy can do any better with his solutions." But he acknowledged that the economic issues "are likely to become more significant" as election day approaches.

Evidence for that likelihood is found in the comments of Claire Rumpel, Democratic national committee-woman for Minnesota, site of one of Carter's caucus victories this year.

"Inflation has been an unfocused issue until now," she said this week, "but our farmers are at the point where they need to buy their seed, and they can't afford to sell last year's crop, the way prices are. They have to borrow, and the interest rates are so mammoth they don't know what to do. And that affects our entire economy."

National Chairman White says, "The general comment I get from Democrats everywhere is that we've got to do something about the economy. This year, people seem to be voting on the candidates first and the issues second. But if the Republican and Democratic candidates are equally acceptable, then obviously the economy will be the major, major issue -- and probably decide the election."

In his contest with Kennedy, Carter has benefited from the overwhelmingly favorable judgments on his personal character, honesty and coolness in crisis -- three elements where voters voice widespread doubts about the challenger.

Robert S. Strauss, Carter's campaign chairman, observed that these character traits may also sustain Carter in the general election. "Once people are for a man, they want to see him through," he said. "It's a tremendous psychological factor."

But Strauss also noted that "this is a strange year, and the voters are a little more transient. There are a great many intervening events that can change things overnight."

Ed Campbell, Democratic chairman in Iowa, where Carter won his first major victory of the year, said the mood among Democrats at last weekend's county conventions was that they were "still willing to go along with him [Carter] and see if he can get on top of some of these problems. But I'd have to say the economy is awfully important, and we have a short time to turn it around."

Campbell said there was "virtually no" leftover bitterness from the Carter-Kennedy fight in his state, "and everyone knows we can't have a repetition of 1968 and 1972 if we want to win."

But others are not so sanguine. Pachios said that in Maine, where the governor supported Kennedy and the other top election Democrats backed Carter, "the wounds are significant. People now talk about the Kennedy faction and the Carter faction. We hoped to avoid that, but I guess it was foolish."

There are signs of potentially serious disaffection from Carter in vital Democratic voting blocs. The Harris survey that showed Ford leading Carter gave the ex-president a 16-point lead over Carter among Jewish voters and a 7-point lead among voters under 30. The same survey showed Ford winning one-third of the self-identified Democrats and 39 percent of the self-identified liberals.

White and Caddell both expressed concern that the defection from Carter of the party's younger and more liberal elements might be channeled into a third-party vote if Anderson, as he has occasionally hinted, takes that route at the end of what both believe to be a futile quest for the Republican nomination.

White said he was sure Anderson could raise funds for such an effort from "the liberal activists on our side," and said that if Anderson gained ballot position in such states as California, Illinois, Iowa, Maine, Oregon, Washington and Wisconsin, it could create serious problems for Carter -- even against Reagan.

State Sen. Mark Kaplan, the pro-Carter Democratic Party chairman in Vermont, said that "if the economy doesn't turn around it's going to be very hard for Carter even to hold onto traditional Democratic support. I've gotten calls from two or three county chairmen this week saying that some of their regular workers are planning to go to the Republican caucuses this year and vote for Anderson. That's something new."

A final concern expressed by Strauss and others in the Carter organization is that the president's self-imposed exile from the campaign trail since the hostages were taken last November could be a problem for Carter in the general election.

"You're just not as sharp without practice," Strauss commented.Another high campaign official recalled that, after his six-week layoff following the Democratic convention in 1976, Carter "took a whole month to get his legs under him, and meantime he was stumbling all over himself. Now, he hasn't been on the road since October, and he's got to be rusty as hell."

All these concerns may come to nothing. But they explain why Caddell said, "The real dynamics of this campaign have almost no relationship to the surface events that show Carter moving from triumph to triumph."