James B. Edwards, the Reagan administrationhs frequently embattled secretary of energy, is "seriously considering" running again for the governorship of South Carolina which he held for four years, according to Republican sources close to the secretary.

Edwards, a 54-year-old dentist who has been described by administration officials as being over his head and out of his league, said he had no plans to leave the administration and had not decided whether to seek another term in the office he held from 1875 through 1978.

Though no one will say it publicly, some White House officials would like to see Edwards leave and would cheerfully support a candidacy for public office to obtain this result.

The South Carolinian was appointed largely because President Reagan heeded the protests of Sen. Strom Thurmond (R-S.C.) and other southerners who complained the South was being neglected in the appointments process.

Edwards has had difficulty organizing his department, and key jobs remain unfilled. He clashed, and lost, with Office of Management and Budget Director David A. Stockman over the revival of a nuclear reprocessing plant. He angered White House officials when he failed to show up for a meeting with President Reagan and coal company officials when a national coal strike seemed imminent.

Despite Edwards' denial, a report circulated through the White House yeasterday that Edwards would be the first Cabinet officer to leave the administration. The idea received encouragement in South Carolina, where influential Republicans believe Edwards would be the strongest GOP candidate in 1982 against incumbent Democratic Gov. Richard W. Riley.

Dr. George Graham, a Spartanburg dentist and close friend of Edwards who is chairman of the South Carolina Republican Party, said in an interview that there is "a good possibility" Edwards will run and that he has encouredged him to do so.

An informal group of leading Republicans who support the idea of an Edwards candidacy met last night in Columbia and it is expected that they also will urge Edwards to run.

Edwards yesterday tried both to discourage the report that he was leaving and keeping his option open to run for governor. He said he had been encouraged to run but believed he was "finally getting hold" of the agency he heads and wanted to complete the job he had taken.

"We're enjoying this job right now and I don't plan to leave anytime soon," Edwards said.

At the same time he emphasized "I am not saying that I am not running for governor" and said he had until the end of the year to decide. Filing date is April, 1982, but Graham said the South Carolina GOP would like to know sometime this summer whether Edwards will be a candidate. In the past, he has surprised supporters in both directions, running for governor in 1974 when he was not expected to and declining to run for the Senate in 1980 when many were convinced he would.

Edwards, one of the few public officals to support Ronald Reagan in his unsuccessful 1976 race against President Ford, left public office in 1978 to resume practice of dentistry.

"It's so satisfying to get my fingers back in the saliva," he said at the time.

The Energy Department has been less satisfactory, according to a number of reports. Edwards lacked knowlege of energy issues and he was, at first, surrounded by officials whom one administration official frankly judged as "on the short side of competence."

Nor did his style seem to fit demands of the department. He left town without telling White House officials where he would be on one occasion, delaying the announcement of Reagan's decontrol of domestic oil. A White House briefing for reporters in which he predicted -- accurately, as it turned out -- that oil prices would level off after initial rises was widely regarded as a disaster.

But Edwards in South Carlina earned a reputation as a dogged and earnest politician who is not the kind of man to quit under pressure. In his interview yesterday he expressed a determination to finish the job and a belief that he is now doing better than at first.

It would not help Edwards as a candidate to return to South Carlina with the reputation of having failed at Energy, particularly if he runs against Riley.

"Jim has a problem," said a friend yesterday. "The good news is that Edwards has recognition ratings in the 70s and approval ratings in the 60s in South Carolina. The bad news is that Riley's recognition is in the 90s and his approval in the 70s."