The special counsel to the House subcommittee investigating how President Reagan's 1980 presidential campaign obtained Carter White House documents said yesterday that the FBI has finished screening campaign documents at Stanford University's Hoover Institution, but that the subcommittee will scrutinize additional documents.

With the investigation in its 10th week, attorney James Hamilton refused to reveal what he has found, but said, "What we've seen so far warrants further investigation."

Meanwhile, an FBI source said yesterday that the bureau is conducting a second round of interviews of Reagan campaign officials to resolve inconsistencies and questions arising from their initial interviews. The source would not say who is being interviewed.

One of the major contradictions revealed so far is between White House chief of staff James A. Baker III and CIA Director William J. Casey. Baker has said he received Carter briefing papers for the 1980 presidential debate from Casey, while Casey has said he does not recall seeing the papers or handing them to anyone.

Baker has told investigators that he received the briefing materials from Casey several weeks before the Oct. 28 debate and has provided the FBI with detailed documents listing his telephone calls and meetings during the period in question, according to sources.

Hamilton said yesterday that the subcommittee investigators have not talked to Casey and Baker.

"I think the best way to do an investigation is to do the spade work before you talk to the principals involved," he said, adding that he has had "fairly lengthy communications" from Casey and Baker and "there does appear to be some conflict."

Casey, who directed Reagan's 1980 campaign, also has denied reports of an intelligence-gathering operation that sought information and documents from inside the Carter campaign.

Hamilton said he is continuing to pursue a Washington Post report that federal investigators recovered a memo in the Reagan campaign files from aide Max Hugel to Casey that conveyed what one source said was a "strong, unavoidable inference" that such an operation was receiving information from someone working for Carter.

After the election, Casey hired Hugel to run CIA covert operations, but Hugel resigned in 1981 after questions arose about his stock and business dealings.

Justice Department officials have denied the existence of such a memo. An FBI spokesman yesterday refused to comment on the matter. Hamilton said, however, "We are pursuing that story. It's an interesting one to look at."

Hamilton said it is too early to determine whether the investigation being conducted by the House Post Office and Civil Service subcommittee on human resources under the direction of Rep. Donald J. Albosta (D-Mich.) will find a violation of federal law. He said the subcommittee also will consider whether legislation is needed to tighten ethics laws.

"What we're all looking into is whether--partially out of the National Security Council--there was a leak or leaks that passed material to the Reagan-Bush campaign, classified information," Hamilton said, refusing to reveal whether any classified information was found in the Hoover files.

"If the stuff came in over the transom, then you've got one level of seriousness. If there was a conspiracy, if money changed hands, then you've got something else. As yet, we don't know what we're going to find," he said.

Hamilton, who was the assistant chief counsel to the Senate Watergate Committee, said there are similarities between this case and Watergate, but "I don't think they should be overplayed." He added that in this case the "chief villain" could have been inside the Carter White House.