Charlie Daniels is hunched over the jump set of a limousine speeding from Columbia, Md., to a midafternoon pool party hosted by Jimmy and Rosalynn Carter at the White House.

Strung out around him are four other limousines carrying half of the crew of 50 that is accompanying him on his current tour. They've brought their swimming trunks, though no one is quite sure what a White House pool party entails. Daniels, who at 43 is built like a 200-pound teddy bear, has left his trunks at his hotel. "I'd feel kind of irreverent going swimming at the White House," he says in a classic and comfortable Tennessee drawl.

Daniels could use a cooling dip: He's still hot under his blue collar from a little incident on a morning television show in Baltimore. This fiddle whose fiery leads have pushed his country-rock-bluegrass group to platinum sales had just been told that he's no good. He was read an article by the Baltimore News-American's rock critic that called his group one of the 10 worst rock bands in America ("We're not even a rock band," says an incredulous Daniels) and also questioned his abilities as a fiddler.

"My gosh ," he blurts out, shaking his head slowly, "I crawled out of a 110-degree tobacco field. I can go back again if I have to. I can work with my han ds to make a living. I want to play music, but I'll be damned if I'll take that guff off some SOB who's telling me -- who's got t-w-e-n-t-y s-e-v-e-n years of equity in this business -- that I'm a lousy fiddle player."

Daniels, who is known to m illions as the fiddler who beat the devil down in Georgia, plans to take out an ad in the New-American challenging the critic to a talk-show debate. But he needn't worry too much. One of his recent albums was titled "Million Mile Reflections" -- and there have been many since he left his native North Carolina to spend three years (late 1958 to early 1961, with a year-long return stay in 1963) on the Washington beer hall circuit. "Skull orchards," he calls it, laughing. "I sold a lot of beer in this town." While honing his skills as a guitar player and fiddler, he lived near Link Wray, and between "Rumbles" they'd play long, hard-fought games of Monopoly.

He played with Washington bands like The Rockets and The Jaguars ("sometimes I'd put my name in front, sometimes I wouldn't"), and then moved down to Nashville in the mid-'60s, where he ended up playing on sessions for folks ranging from Flatt and Scruggs and Ringo Starr to three albums' worth of Bob Dylan. "I was very impressed working with Dylan," Daniels recalls. "He wanted people who were capable of playing his music in their own way. He wanted you to bring something into a session. Of course, playing with him didn't hurt me professionally either."

Apparently not: He won eight national musical awards in 1979, and sold 3 1/2 million albums in that year alone ("I'd rather sell records and get bad reviews," Daniels says, moving back to thoughts of a Baltimore critic. "How'd he get the job, anyway? I could show him some bad bands. . . .")

In the last 10 years he's recorded 11 albums and had a half-dozen No. 1 singles. The first was "Uneasy Rider" in 1973, followed in 1976 by "The South's Gonna Do It Again" (not a big hit up north). Last year it was "Devil Went Down to Georgia," and most recently the patriotic morale-raiser, "In America." Daniels is also featured prominently in the "Urband Cowboy" film, putting in a long appearance than any other performer except Mickey Gilley.

Yesterday was not Daniels' first trip to the White House. He was one of the earliest musicians to suport Jimmy Carter, starting in the 1976 primaries. "I got some press clips and became aware of what he stood for. I just felt he was a damn good man." Daniels played three Carter campaign benefits in 1976; he has done one this year already, and expects to do at least one more. In the last four years, he's visited the White House three times for varied functions; he also performed at the 1976 inaugural -- the first time rock music had been heard in those normally somber affairs.

Before "In America" was released, Daniels sent an advance copy to the president. "He sent me back a nice . . . nonpolitical . . . letter in which he spoke of the musical value. He said it was a good song, and he appreciated having an advance copy." Carter also took time out earlier this year to phone Daniels while he was recuperating in the hospital from a broken arm seriously fractured in an accident on his Mt. Juliet, Tenn., farm, where Daniels raises quarterhorses and Hereford cows. "He's just as nice as he's ever been," Daniels says in appraising the changes in the president over the four years he's known him. "It's worn him, though. He looks older than he did. It's a damn shame he gets belted around the way he does. It's made him harder . . . maybe he needed that."

Daniels is no stranger to well-known figures. He recently met "Dallas" TV star Larry Hagman ("J.R. met me ," he chortles) in Opryland. "Minnie Pearl gave him one of her hats with a price tag on it and I gave him a western hat [Hagman collects hats]. And you know what he was wearing? A hat made entirely from rattlesnakes -- with rattles on the back." This gets a big laugh from Daniels, as well as his wife, Hazel, and 15-year old son, Little Charlie, who are going with him to the White House.

He also feels that his appearance in "Urban Cowboy" has probably helped the film more than the film has helped him. With an average of 200 shows a year around the country over the last eight years, he's probably been seen by more people live than will ever see him on the screen until "Cowboy" makes it to television. "Some of our hard-core fans probably only went to see it because we're in it." In the movie, as in his live performances, Daniels tends to keep his eyes closed, and shades them further under his big-brimmed cowboy hat. "I always keep'em shut," he says. "It's part of my concentration. I've got a one-track mind. I just close my eyes and do what I feel I ought to be doing. Anyways, I don't figure people pay to see me, they pay to hear me."

Despite his national success, Daniels remains a very gentle, friendly, open man whos main concern, aside from spending all of his free time with his family, seems to be to "beat last night's performance. I've been playing 'The Orange Blossom Special' for 22 years. And I still enjoy it every night, nobody in the audience will have a better time than I will.

"What I am is a simple, working class person. I came from laboring people, farming stock, people who made a living with their hands, who got up before daylight and worked until dark. I'm proud of that. That's the people I'm doing my music for."