The small white sign taped inside the front door of the vacant house on Second Avenue here merely says the doctor is out and won't be coming back.

There is no explanation offered for why Dr. James C. DeWitt suddenly gave up his practice in this isolated eastern Colorado farm community, or of the double-edged tragedy his brief stay here has caused.

And there is certainly nothing on the note to indicate Dr. DeWitt's hasty departure has anything to do with a larger problem that small towns like this one increasingly face when they deal with what some medical officials refer to as "defeated doctors."

These are doctors with emotional or sexual problems, alcoholic doctors, or physicians, like DeWitt, whose history includes drug addiction.

"They want to make it and they try but they just can't," said Dr. Stephen Barnett, head of the federally funded Mountain Plains Outreach Program, a Denver-based organization that tries to help find doctors for desperate rural towns such as Julesburg.

"The ones we have come across who are defeated doctors start to drift," said Barnett. "They work someplace for six months or a year and then the roof caves in and they move on to some place else."

According to estimates by the American Medical Association, about 4 to 5 per cent of the doctors in the United States can be said to have "some kind of impairment." That means, said an AMA spokesman, they have drug, alcohol or emotional problems.

The defeated doctor problem has reached such proportions that 25 states now have what are known as "sick doctors laws." The laws generally provide for a medical committee review of doctors who have alcoholic, drug-use or emotional problems. If the committee decides there is trouble a doctor's practice can be curtailed or suspended.

Recruiters like Barnett or Mario Manecci, of the federal National Health Service Corp. which also provides doctors for rural areas, and state medical licensing officials like New Mexico's Dr. Robert Derbyshire rattle off case after case of rural towns who have sought medical help and ended up with defeated doctors.

"A lot of these places are willing to overlook almost any problem, or they just don't know how to go about identifying a doctor's weakness before it is too late," said Derbyshire, a past president of the U.S. Federation of State Medical Boards. "These are very vulnerable places."

The most serious case, according to Rex Higley, director of Nebraska's bureau of examining boards, involved an unlicensed man who posed as a doctor several years ago and answered ads in Nebraska, Colorado and Montana, moving each time after his lack of credentials was exposed. The man was finally jailed after several patients died at a Montana veteran's hospital where he was practicing, Highley said.

In Chappell, Neb., a few miles from here, residents placed an advertisement for a doctor several years ago. They finally had to hospitalize the doctor they recruited after one of his patients noticed the doctor injecting himself with a drug beneath his desk during a medical conversation.

The story of Dr. DeWitt's stay in Julesburg began in 1975 after one of the town's two physicians died and the second became ill with cancer.

The town formed a search committee and put ads in newspapers in Los Angeles, San Francisco, Kansas City, New York and Boston. Without a new doctor, Julesburg faced the possible closure of the county hospital and a retirement home.

Members of the Julesburg search committee said they had two "screwballs" who answered the ads and were flown out - at the town's expense - for a visit. One, they said, graphically described his daughter's sexual habits during a luncheon with residents. The other turned out not to have a license.

The arrival of DeWitt, a 55-year-old physician who answered an ad in the Boston Globe, was greeted enthusiastically. The Julesburg Advocate, the weekly newspaper, ran a front-page story and the Chamber of Commerce raised $3,000 to help DeWitt open his practice. Only after his arrival did residents here learn that DeWitt had been a narcotics addict in the 1960s.

In a short telephone interview this week DeWitt acknowledged his addiction in the 1960s. But he said he overcame his habit and more recently had worked with other addicts in a Massachusetts hospital. He even wrote a book about his experiences. It was called "Addict - A Doctor's Odyssey."

State medical officials and federal narcotics officials refused to discuss the details of DeWitt's case. But knowledgeable state officials said that irregularities were reported in his drug prescription practices in Julesburg.

Last month, according to his state attorney general's office, DeWitt voluntarily agreed to turn in his Colorado medical license in return for an agreement from the attorney general's office to drop its investigation into the affair. Under the agreement state officials promised not to reveal details of DeWitt's problem.

The doctor, who was preparing to move out of his small garden apartment this week, refused to meet with a reporter. But during the telephone interview he said he knew of other doctors with problems like his own. "You hear about doctors' drug problems but most of the time they don't leave it in the open," he said.

DeWitt said he would move to another state. "I just want to go somewhere and start again where no one knows me," he said. "I'm tired of fighting this thing over and over." He declined to say where he would go next.

In additiion to the collapse of DeWitt's life here, his abrupt departure has shocked this town.

But Ronald Wilkins, publisher of the Julesburg Advocate and a member of the committee that recruited DeWitt, said he supported the doctor and that his arrival was necessary for the town's economic survival. "He was very good and we needed him," Wilkins said.