At the ripe old age of 27, Patrick H. Caddell is learning the bittersweet taste of fame.

The years ago, when he was a precocious student at Bishop Kenny High School in Jacksonville, Fla., it was a heady experience to show a roomful of state legislators his solution to a redistricting problem that had baffled all of them, or to go on television station WJXT and accurately predict the outcome of local elections.

Five years ago, while a senior at Harvard, he was regularly on national television, explaining to a mystified country how the candidate for whom he was polling, Sen. George McGovern (D-S.D.), was winning all those primaries.

His own polling business was profiting from the publicity, and journalists twice his age were eagerly recounting the beguiling story of this political wizard, who had to interrupt his work on the crucial California primary in order to fly to Cambridge and pass the swimming test Harvard requires of its prospective graduates.

When Caddell turned up last year as the pollster for another long-shot contender, Jimmy Carter, it was regarded as a political coup for Carter, not a business coup for Caddell.

And now, as the President's pollster, ensconced in a newly opened, top-floor business office two blocks from the White House and a Georgetown bachelor pad, Caddell is receiving more attention than ever.

But this time, he is enjoying it less. "I understand," he said, "that people snipe at you . . . that there is a matter of jealousy, and so on. I'm willing for people to raise any legitimate questions they want to about me, short of being told I can't do business.

"But no one has ever come up with a legitimate charge of a conflict of interest, and to constantly have the question raised without any grounds is certainly not helpful. No one likes to have his integrity questioned."

The controversy that has enfolded Caddell in the past month focuses on two fronts. Ever since a political strategy memo he wrote Carter last December was leaked to the press in early May, the political Caddell has been criticized by liberal Democrats as an apostate guiding the President to a neo-conservative policy line that contradicts his own professed principles.

And as a private businessman with privileged access to the Oval Office and an expanding list of corporate clients with a vast stake in government policy, Caddell has been challenged on the conflict-of-interest charges to which his own comments referred.

On both the political hypocrisy and conflict-of-interest charges, Caddell pleads innocent. Neither the senior White House aides nor his colleagues in the polling business believe he is deserving of censure.

Liberals, including McGovern, profess to having "no hard feelings" toward Caddell and say they still value his friendship and professional services.

And yet the nervousness Caddell's own comments betray is shared by those with a great personal stake in his future. "We're obviously concerned about the stories," says John Gorman, chief partner and "inside man" in Cambridge Survey Research, Inc., "but aside from going out of business, there's not much we can do."

Fred Schultz, the Jacksonville businessman who befriended the young Caddell and lent him the money to start his business, says:

"Here he is, a young man who is thrust into the limelight and has a great opportunity to become successful in terms of fame and fortune. Obviously he wants to take advantage of these things as much as he can, and yet he doesn't want to overstep any bounds. My advice - and we've talked about it - has been: when in doubt, don't do it."

It is not clear whether the private concerns or the published comments are slowing Caddell.

He said in an interview that while he has ignored "feelers from sone people" he suspected were interested in his influence rather than his date, "I've not turned down anyone per se." Business has expanded to the point that, according to Gorman, there are 50 per cent more full-time employees in the Cambridge headquarters than there were two years ago.

Among more than 20 clients that pay $20,000 a year for the quarterly Cambridge Report, a massive inventory of public attitudes on political, social, and economic issues, are such blue-chip firms as Westinghouse, Exxon, Citicorp, Merck, Atlantic Richfield and Sears Roebuck.

Caddell shares office space (and a Georgetown house) with Jerry Rafshoon, the Atlanta advertising man who handled the Carter campaign and also serves as a presidential adviser. The two are working together for Henry Howell in the Virginia gubernatorial primary, Mario Cuomo in the New York mayoral race, and are exploring the movie business together.

They are handling audience surveys and marketing strategy for Francis Ford Coppola's forthcoming Vietnam was film, "Apocalypse Now," and, according to Caddell, are "studying other possibilities."

The prospering young businessman in the conservative-cut three-piece business suit is a far cry from the "somewhat disheveled" collegian introduced to Gary Hart, then managing McGovern's campaign, in Miami in 1971.

Hart, now a senator from Colorado, was impressed by the youngster's "sense of moods and attitudes," so much as by his willingness to work for a longshot like McGovern. He agreed to pay him $500 for a quick survey of New Hampshire.

McGovern says that Caddell "found an uncanny resemblance between what the public was telling him and what I was saying," and came quickly to lean on his pollster's insights.

Even now, despite Caddell's switch to Carter and McGovern's role as the loudest liberal critic of Carter's domestic policies, the South Dakotan finds it hard to fault his young friend. "I think Pat's basically a humane, liberal-minded man, a man of compassion and liberal spirit," McGovern said.

There was, however, some awkwardness during the period in 1975 when McGovern was considering a second try for the presidency and Caddell was increasingly drawn to Carter. Caddell says he told McGovern "I would, of course, help him" if he ran again for President, but felt he had almost no chance to win. He was, admittedly, relieved when McGovern released him from "that obligation" by deciding not to run.

Caddell himself concedes that "my view of politics changed enormously" between 1972 and 1976, as a result of his shift of focus from the Vietnam war issue to the broader question of eroding public confidence. But his critics see a strong element of opportunism in his McGovern-to-Carter switch.

Alan Baron, editor of a liberal newsletter, says of Caddell: "He has an amazing ability to tell his clients that, deep down, the American people believe what that client wants them to believe. I was with George McGovern last year when Pat told him the American people were way to the left of Ted Kennedy and ready for really radical change. That's sure not what he's telling Carter."

Charges of hypocrisy are rejected by many others who know Caddell. They note that his alliance with Carter was a natural one for a Southern liberal, long fascinated by the emergence of a "new generation" of political leaders in his own region, and long preoccupied by the danger he saw in George Wallace's mixture of populism-and-racism.

But liberal criticism of Caddell increased after publication of his De-Cembermemo to Carter, focusing on inflation-conscious white-collar, professional and college-educated voters as they key tagets for the next four years and warning that Carter's real opposition would come from an "antiquated and anachronistic" group of "traditional liberal" and "Young Turk" Democrats.

While defending his thesis, Caddell is plainly sensitive to the way in which some of what he calls "shorthand phrases" have been interpreted.

He says it has been "excruciatingly painful to have it portrayed" as personal criticism of people who are "clients and friends of mine and who have stood up on tough issues."

"What's antiquated," Caddell says, "are traditional ideologies, not people." McGovern and Hart, mentioned by name in his memo as examples of those who might cause Carter problems, both say they have told Caddell to forget the incident and that they will welcome his services in their next campaigns.

More worrisome are the charges of profiteering and potential conflict-of-interest that began last year when New York Times columnist William Safire criticized Caddell's contract to provide polling services for the Saudi Arabian government.

That contract has lapsed, but the same criticism has recurred in recent newspaper columns and magazine articles.

Caddell's answer is that a conflict of interest would arise only "when you try to help someone or lobby for someone, and we do not do that."

"Our obligation is to an individual client to give them the best information and results we can. It's a professional relationship, and the key is to maintain your objectivity and impartiality."

Caddell's stance is generally backed in the polling profession, where it is customary for firms to handle both political and business or interest-group clients at the same time.

"If a pollster is going to stay in business," says a Caddell competitor, William Hamilton, "he has to do that sort of thing."

But, as Caddell concedes, the problem becomes acute when the political client is President of the United States. Caddell is unquestionably a White House insider. He carries a White House pass, even though he is not on the payroll, and regularly attends the Wednesday strategy meetings conducted by Carter's senior staff aide, Hamilton Jordan.

He has billed the Democratic National Committee for $96,000 so far this year for the four or five polls he has taken on behalf of the President. His memos to President, like those of Rafshoon and Atlanta lawyer-adviser Charles Kirbo, are not screened by anyone on the staff. Senior aides Jordan and Jody Powell affirm that the President - an inveterate student of the polls - still leans heavily on Caddell's advice.

Powell said in an interview that Caddell "very seldom" advises on substantive policy, but he concedes - as does Caddell - that everyone from the President on down has given some thought to the proprieties of Caddell's role. So far, the decision has been to let Caddell himself decide when the interests of his private clients might cloud his dealings with the President.

"It's hard to establish a rule, but there's obviously got to be a line somewhere" Powell says. "If Pat or anyone else with a private client is playing a role in (policy) matters that directly affect that client, it'd be wrong. But I don't even know who his clients are."

Caddell says he has no objection in principle to making public a full list of his private clients, but so far he has not done so.

His attitude is that of a man who wishes that his position were not a matter of so much curiosiy. "At this time," he said, buttoning his vest at the end of a recent two-hour interview, "I'd just as soon be anonymous."