For Jimmy Carter, his inaugural address on Jan. 20 will be the start of a new career. For Patrick Anderson, his chief speech-writer since last May and principal collaborator on the inaugural, it will be the finish.

Anderson, 40, author of the best-selling novel, "The President's Mistress," confirmed yesterday that he will not be going to the White House with Carter as head of the speech-writing staff as planned.

Replacing him in the job will be his deputy from the campaign, James Fallows, a 27-year-old Rhodes scholar and former president of the Havard Crimson board of editors.

Explanations of the last-minute switch in plans varied. According to Anderson, who was in Plains, Ga., yesterday to confer with Carter on the inaugural address, he decided himself last Sunday that he preferred to pursue his own writing career rather than follow Carter to the White House.

Others close to the situation say there was a "conflict of personalities" and some sensitivity on the part of Carter insiders to the publicity given Anderson's mildly "gamy" novel being the work of Carter's "speech-writer."

Anderson said yesterday that Carter had promised him "a great deal of cooperation" if he would write a book about the campaign and the first year of the Carter administration. He said he was seriously considering the project.

He has just put the finishing touches on a book on Carter's campaign and pre-campaig speeches, the proceeds of which will help launch the Carter presidential library fund.

Anderson is the second senior Carter aide from the campaign to fall by the wayside before the White House doors open. Greg Schneiders, who was slated to be Carter's appointments secretary, removed himself from consideration last week when an FBI report showed he had several outstanding legal and financial problems.

Anderson's wife, Ann, is going ahead with her assignment as Rosalynn Carter's deputy press secretary.

Anderson joined Carter's staff last May after writing an admiring profile of him for the New York Times Magazine. He replaced Robert Shrum, who quit, after only brief service, with a public blast at Carter's "manipulation and deception" of the voters.

Anderson was a principal contributor to Carter's acceptance speech, his general election kick-off speech in Warm Spring, Ga. and other "theme" speeches of the campaign.

A native of Redlands, Calif., Fallows was graduated from Harvard in 1970, then spent two years at Queens College, Oxford, on a Rhodes scholarship. From 1972 to 1974 he was on the staff of the Washington Monthly, writing profiles of media figures, among others, as well as an article analyzing why and how he and his Harvard friends had rationalized avoiding the draft in the Vietnam war period while less-educated comtemporaries went off to war.

Fallows, who has lived in Austin, Texas, as a free-lance writer for the last two years, said yesterday he had decided to take a job under Stuart Elizenstat on the White House domestic policy staff when Carter called him last week to offer the chief speech-writing job. Fallows said he would not comment on why Anderson was not removing into that post.

Fallows said he did not expect to do any work on the inaugural address, but was taking over from Anderson immediately on interviewing candidates for six or seven other positions on the White House writing team.

Anderson said, "I have a career of my own which is going well, and I decided that best kind of relationship to have with Carter is like (pollster) Pat Caddell or (television producer) Gerald Rafshoon."

Caddell and Rafshoon are not going on to the White House staff but will continue to work on contract for Carter and the White House.