FROM THE TIME I left the White House as President Lyndon Johnson's staff assistant for domestic affairs in January 1969, I knew that if I ever returned to government service. I would like to be secretary of health, education and welfare. But in light of Jimmy Carter's attacks on Washington insiders during the 1976, I never thought he would offer me the opportunity -- until he chose Fritz Mondale, a personal and political colleague of mine since 1965, as his running mate.

Then, after the Democratic National Convention, Patrick Anderson, a novelist and the first Carter speechwriter, called me: "Jimmy wants to talk about the family at his first postconvention appearance in Manchester, N.H. He'd like your help on the speech." The next day Mondale introduced me to "Gov. Carter" over the phone.

"Too much federal policy is anti-family," Carter said. "I want to make it pro-family. The welfare system breaks up families and we've got old people living in sin because they lose Social Security benefits if they get married . . ." Carter left mondale and me on the phone to talk about the speech.

It's really important for you to help on this. I want Carter to get to know you," Mondale said, signaling as a political friend can the potential ahead. Pat Anderson called two days later to say, "Jimmy wants to announce in his speech that he's asked you to do a report on how government programs affect the family." Carter also wanted to see me in New Hampshire.

I greeted Jimmy Carter in Manchester on Aug. 3. We rode together to the shopping center where he was to speak. When we got in the car, an aide showed Carter that day's issue of the Manchester Union Leader, William Loeb's right-wing paper. Two front-page stories excoriated Carter. His face tightened with a hint of controlled rage as he read them. I thought to myself, this was something we would never do with Lyndon Johnson just before he was to deliver a speech; his temper would almost certainly overwhelm his discretion. As I looked at Carter, his eyes were so calculating I assumed he was more self-controlled.

When Carter ad-libbed for 15 angry minutes that this was likely to be one of the dirtiest campaigns in history, that the Republicans would stage an "almost unprecedented vicious personal attack" on him, I was not sure whether he was truly angry or had decided to use this as a way of positioning himself for the future.

He then recited his prepared text on the family, with a "pledge to you that every statement I make, every decision I make will give our families a decent chance to be strong again" and the announcement that I would be his adviser on the family and make a report to him.

Carter and I lunched alone at a house adjoining the Manchester Hotel. Only one lunch was delivered: steak, vegetables and a big bowl of fruit. Carter pressed me to join him, cu the steak in half and divided the single portion of vegetables.

We talked for almost two hours. I was struck by how religious he appeared and wanted to appear, how confident he was of defeating President Ford, how innocent about the difficulty of achieving massive reform in the national government. When I suggested the increased difficulty in moving Congress in the wake of Vietnam and Watergate. Carter recounted an experience with the Georgia legislature.

"When they didn't cooperate with me, I went across the state, speaking to their constituents. They refused to pass a consumer-protection act until I did that. If the Congress doesn't move, I'll get the American people to move them." Even though I was to underestimate the intractability of the Congress, I knew he was overconfident.

As Carter questioned me, he struck me as superficially self-effacing, but intensely shrewd. He did not disguise his intention to make honesty and competence the issues, to target on Nixon's scandals and Ford's bumbling, and to run as a nonpolitican and Washington outsider.

His questions about Democratic establishment figures were as penetrating as they were persevering. They reflected his early concern about the loyalty of party chairman Robert Strauss' ambition and zest to be at the center of the action would in the end motivate him. In a colloquy about Cyrus Vance in which I said that Vance was not only brilliant but has "as much integrity as any person I've ever met," Carter asked, "I don't doubt it, but is he tough enough to be secretary of state?"

Carter turned to the problem of the family. He expressed his concern about the family with the same sincere conviction he exhibited in discussing his concern about the Catholic vote. He urged me, in preparing the report on the family, to consult widely with Catholics and look for an appropritate forum for a campaign speech.

Carter talked about the Great Society programs of Lyndon Johnson and my work in those years, but much of his immediate interest in me was prompted by the fact that I was a Catholic. It was the first time in my life I had been singled out (either for favor or discrimination) for that reason, but I was so interested in the outside possibility of the HEW post that I never thought twice about it at the time.

After the election, Mondale told me he was urging Carter to name me to a Cabinet post, preferably HEW. In mid-November, House Speaker Tip O'Neill told me Carter was considering me for a major post.

"I spend the whole damn ride from Carter's house to the airport this afternoon talking about you," he said when we bumped into each other at Duke Zeibert's. "He kept asking me questions about you. He mentioned two or three jobs. When he mentioned HEW, I told him you'd be a great secretary of HEW, but I said, 'Mr. President, Joe isn't going to take a job like that. He's served his time with Lyndon Johnson. He makes a fortune as a lawyer in Washington.'"

"Tip," I exclaimed, "I'm interested in that job. I'd like to prove that HEW can be run, that those Great Society programs can work."

"You've got to be crazy," O'Neill said, then recovered quickly. "Well, I recommended you and, you poor guy, you may just get it."

The first time I saw Carter after the election was when he interviewed me on Dec. 7 at the governor's mansion in Atlanta. Half of the hour-long interview was conducted with Hamilton Jordan and Charles Kirbo present; the rest was alone with the president-elect. By this time, Carter knew a great deal about me.

He asked me two questions that I in turn asked almost everyone I interviewed for HEW: "What is the most difficult thing you have ever done in your life?" and, "I've got a folder full of your good qualities, but what do you think your weaknesses are?"

"Most difficult?" I thought for a moment. "Working for Lyndon Johnson. But it was also the most satisfying. As for weaknesses, I suppose it's that I've never run anything larger than a law firm."

Carter asked if I had any questions. "Only one," I responded. "Will I have the ability to pick my own people?"

"Yes. Many are presidential appointments, of course, but barring a crime, or some serious embarrassment in an FBI check, you can selected your own people. I intend to keep my promise of Cabinet government to the American people."

I was pleased. Robert McNamara, when John Kennedy urged him to become secretary of defense, had asked only for that. He later told me it was the most important request he had made.

By Dec. 21, Carter had named all his Cabinet except secretary of HEW. Despite stories in the press, I had still not heard from him. Just as I was leaving my home for the office, the phone rang. "This is Jimmy Carter. I'd like you to be my secretary of Health, Education and Welfare."

"I thought you'd never ask, Mr. President," I quipped, and promptly added, "It would be an honor and a privilege."

Abe Ribicoff called. The Connecticut senator had been HEW secretary under President Kennedy. "I wish you all the luck in the world and it still won't be enough for that job. In less than a year, the only way I could keep my sanity was to ask them to change the colors of the briefing books each day. You'll read more briefing books in a month than you've read in the last eight years." I laughed.

"I'm serious," Ribicoff continued. "The job is truly impossible. When you walk down the corridor to your office, just see how many pictures of former HEW secretaries hang on the wall. You're constantly caught between the president and the constituencies you represent. You can't win, because you represent the poorest people who never get their fair share, and every president has budget problems. But look, I know you, you're making is worth it."

My first impressions of Carter's presidential style came at Sea Island, Ga., during his initial meetings with the Cabinet. The Cabinet stayed at the Cloister, a luxury resort; the Carters at the Musgrove Planation on St. Simons Island 10 minutes away.

I was stuck by the ostentatiously non-presidential ambience of both the new president and his associates. Carter brandished informalities and religion. He slouched in a sweater and jeans, spoke softly, constantly appearing to defer to comments by members of the new Cabinet, especially Cyrus Vance.

He prayed before meals, exuded Fundamentalist intensity, invoked the name of God frequently. In each of our rooms at the Cloister was a small book of religious poems written by LaBelle Lance, the wife of his friend Bert, who had been named director of the Office of Management and Budget. I attributed much of this to Carter's born-again Baptist beliefs and suppressed my Northeastern Catholic discomfort at such public displays.

The odor of naivete perfumed those two days off the coast of Georgia. The new president evidenced little sense of the complexities of governing. Except for Stuart Eizenstat, who had been a junior aide under Johnson and had worked on Hubert Humphrey's 1968 presidential campaign, and Jack Watson, who had conducted a wide-ranging transition study, Carter's staff seemed naive to a fault and appeared to believe the anti-Washington rhetoric that had carried Carter to the White House.

In the meetings, Carter spoke sincerely of his desire to use his presidency "for good," to restore the confidence of the people in their government, to "give them an administration as good as they are."

Hamilton Jordan worked at being the country boy from Georgia, wearing work boots, affecting boredom during much of the discussions. Jody Powell was disingenuously deferential, calling each Cabinet member Mister or Madame Secretary. Watson, the Carter staffer with whom I had discussed organizing the government, was subdued, giving some validity to news reports that he was having his wings clipped by Jordan; Eizenstat was quiet and serious. As I sat at the meetings, I thought that Watson and Eizenstat would have to provide Carter his substantive staff advice.

Bert Lance, who had been Carter's Georgia highway commissioner and banker, was charming, but displayed neither the depth nor motivation required to be OMB director, and the others close to the president evidenced little interest in governing.