Battle lines have been drawn between two of the Reagan administration's most powerful members, CIA Director William J. Casey and White House chief of staff James A. Baker III, over how Reagan political aides wound up with documents and information from President Carter's White House and 1980 campaign.

"It's hard to see how both men can survive this because their stories conflict so fundamentally," said one high-ranking White House official, who acknowledged that the dispute could have far-reaching implications for the future of the administration.

Casey insisted in strong terms yesterday in a telephone interview with The Washington Post that he had never seen the disputed material, would have been suspicious of it if he had seen it and would not have given it to Baker.

"I'm not involved in anything like that," Casey said in the interview. "I was supposed to be the guy who didn't remember, but I can't remember something that didn't happen."

Casey also said he knew nothing about campaign adviser Richard V. Allen's statement that Allen received various "innocuous" excerpts of daily reports from National Security Council staff members. Casey said that Allen worked in a research group headed by campaign chief of staff Edwin Meese III and that he--Casey, the campaign manager--did not see much of the material that came into this unit. Casey's latest comment put him on a collision course with Baker, who has said repeatedly his recollection is that Casey gave him the material during preparation for the October, 1980, debate between Ronald Reagan and President Carter.

Without retreating, Baker yesterday issued a brief statement saying that he was limiting his comments "only to those who are conducting the inquiries."

On Tuesday, according to sources, Baker sought to prevent Casey from giving his views to The New York Times, suggesting that both men withhold comment until the outcome of investigations being conducted by the FBI and a congressional subcommittee headed by Rep. Donald J. Albosta (D-Mich.).

Instead, Casey granted the interview and said that it would have been "totally uncharacteristic and quite incredible" for him to have obtained the material, which he said he "wouldn't touch with a 10-foot pole." He also said that Baker was "remiss" in not telling campaign officials that the materials had been given to him. Casey repeated these statements yesterday, emphasizing that he would have been concerned about "a setup" by Carter strategists.

Casey and Baker, along with White House communications director David R. Gergen and Office of Management and Budget Director David A. Stockman, face questioning by the FBI, probably later this week.

Hundreds of pages of the Carter material were found in Gergen's files. Stockman said in a speech on Oct. 28, 1980, the day of the debate, that his preparation as Carter's stand-in for the debate rehearsal had been aided by a "pilfered" Carter debate briefing book.

President Reagan also may be questioned. Reaffirming what White House spokesman Larry Speakes said while Reagan was vacationing in Santa Barbara, Calif., Meese said that the president would agree to be questioned by the FBI.

The usually affable Speakes bristled yesterday at the daily White House press briefing as reporters questioned him about the differences between Casey's and Baker's statements. Speakes insisted that there wasn't necessarily any difference in what the two men were saying. When reporters persisted in their question, Speakes said of them: "I think you are possessed."

His comment was a public reflection of the growing tension in the White House, where a story that one high-ranking official had once labeled "a one-day wonder" has become an embarrassment for the administration.

The sharp difference between Casey and Baker reopened barely healed wounds between the administration's "Reaganaut" wing, consisting chiefly of longtime adherents of the president, and a group of more moderate conservatives, headed by Baker, many of whose original loyalty was to other Republican candidates or presidents.

Most of the longtime Reaganites served with Reagan when he was governor of California or in previous presidential campaigns. Casey, a New York attorney and chairman of the Securities and Exchange Commission in the Nixon administration, was not one of this group, but he became a favorite of the Californians when they ousted campaign manager John P. Sears during the 1980 race.

In the administration, Casey often has supported the views of national security affairs adviser William P. Clark and Meese in opposition to Baker. He has been rewarded with staunch backing from conservatives, many of whom have also called for Baker's replacement.

Some conservatives reportedly see what one of them yesterday called "a silver lining" to the current controversy, by which they mean an opportunity to oust Baker. If that happened, it is generally agreed in the White House, other moderates associated with Baker would resign, leaving the administration in the hands of ardent Reaganites.

Casey disclaimed any knowledge of such a maneuver, and said he was not involved in an attempt to push out Baker.

Many Baker supporters took this statement at its face value, and some acknowledged that Casey was on the defensive because Baker's original statement had fingered Casey as the source of the documents.

"Casey is trying to save himself, not launch a conservative coup," one official said yesterday.

Regardless of motives, the differing recollections of Baker and Casey have had the effect in the White House of rekindling old divisions among the various factions.

This time, however, the divisions were less clear-cut than in previous staff battles, as officials who were not involved in the campaign tried to stay clear of the controversy.

This was particularly true for Clark, who was a California Supreme Court justice during the 1980 campaign, and presidential assistant Richard G. Darman, a Baker adherent who was a Harvard professor and economic consultant in 1980.

"Maybe we can wind up with Clark as chief of staff and Darman as his deputy," one official joked.

Two other top White House officials, deputy chief of staff Michael K. Deaver and Meese, were central figures of the 1980 campaign but Deaver spent most of his time on the campaign plane serving as Reagan's personal aide. He has not been mentioned as a recipient of any Carter campaign material.

Meese, now the White House counselor, was mentioned for the first time yesterday as the possible recipient of some of the Carter campaign memos. He is considered a strong ally of Casey.

Baker and Gergen have taken the view that the less said about the controversy the better. They have avoided interviews, and their supporters say that they are confident that they will be cleared by the investigations.

Speakes' unwillingness to comment yesterday was based upon Baker's instructions, according to White House sources. One of of them called it "an experiment in damage control."

In line with this decision Baker issued a brief statement yesterday which said: "The president has instructed us to cooperate fully with the congressional and Justice Department investigations. I have been doing that and will continue to do so. Out of concern for fairness and for the integrity of the investigative processes, I am commenting only to those who are conducting these inquiries."