When he first spoke to Jimmy Carter, Marine Maj. Jeffrey Zorn recalls the president telling him, "We have certain similarities in our background. I would enjoy talking about those similarities."

The reference, says Zorn, was to the religious beliefs he shares with the president: Both are born-again Christians. But the two never got around to talking about that again.

After only a few months in a job that promised a brighter military career, a job that took him jogging and fishing with the president of the United States, Zorn was relieved of his White House duties as a military aide and transferred back to a desk at his home base of Quantico.

The reason: Since 1976 Zorn has refused to file a tax return with the Internal Revenue Service. Nor does he intend to.

The decorated 33-year-old officer refuses on religious grounds, saying he has no money to report because he has taken an oath of poverty with his church, the tiny, four-year-old "Third Estate Church" in Prince William County. The congregation meets in Zorn's suburban Viginia home on the Quantico base and he is one of its evangelizing pastors.

"As I became convinced that this is where God is leading me, I took this stand," Zorn, who has refused to divulge either personal or church records to the IRS.

The entire matter, he said in a telephone interview yesterday, is a product of an evil foretold in the Bible. "I'm not saying that the Second Coming is around the corner," he said. "But something definitely is afoot."

Neither the IRS, nor apparently, the Defense Department, sees it quite his way.

Although the IRS will divulge no details in the case, Zorn says his church is being investigated as a possible bogus tax dodge. "We do investigate so-called churches as sham transactions," said IRS spokesman Steve Schneider in Richmond. "Some of these churches founded on certificates from bogus institutes are set up solely to avoid taxes."

The Defense Department apparently isn't taking any chances. When it was notified by the White House that an investigation was under way in April, it ordered Zorn to pack up and go home.

On the morning of his last day at the White House, Zorn said he was told by the chief of the president's military office, Marty Beamon, that "it would be better for everyone concerned that we not have this embarrassment surrounding the president."

Zorn, who served in Viet Nam in 1970 and holds the Bronze Star, is convinced that although he has a master's degree in theology and is licensed to preform marriages in Virginia, the IRS refuses to beliefve his Third Estate Church is genuine. Only about a dozen persons are members, said Zorn, and the closest thing to a steeple is an occasional sparrow that roosts on the roof.

"They [the IRS] are operating under the traditional concept of the church as it is today," said Zorn. "We're practicing under the Apostles' conception of the church as it was 2,000 years ago."

Zorn said he has no beef with paying taxes. "I want to cooperate. We're not a cult and we don't take cocaine or marijuana. If I owe any taxes, I'd be happy to pay them. All I've done is take a vow of poverty.I have been classified as a tax protester."

But he has few kind words for the IRS. His distaste for that agency burns with biblical fire.

The first Zorn knew that he and his church had fallen under the IRS magnifying glass was a cold, rainy night in January. He heard a knock on the door and opened it to find two IRSagents requesting his tax records for 1977 and 1978.

"They came without any notice," he said. "I asked them, 'Have you made an appointment? Is there some reason why you're standing at my door like the Gestapo?' They told me no, this was IRS procedure. 'We like to catch the individual off guard,' they said."

Zorn said that since that night he has made repeated requests of the IRS for information regarding the case but that he has been received with nothing more than a bureaucratic yawn.

What ignites his religious fervor most is the IRS demand that he release church records. This he regards as part of a larger, growing evil, and he can cite chapter and verse from the Bible to condemn it.

Zorn, who says he spends most of his free time looking for converts both locally and across the country, has taken his case to his commander-in-chief.

In May, he wrote Carter, saying that "the Internal Revenue Service is embarked on a campaign of blanket classification of independent, nondenominational churches as 'tax protesters' with the aim of casting doubts about their integrity. Their goal is to control church activities which will aid in the establishment of a state church."

"The Lord led me to serve my country, and as my ability to listen to His voice and be obedient to it grew. He chose more and more difficult tasks for me. Recently, He led me to be appointed as your aide, where I was close to the leaders of our nation and where I might have the opportunity to witness to these great men and women directly, for the glory of God."

The response, from White House Counsel Lloyd N. Cutler, informed Zorn that: "For obvious reasons, it is inappropriate for the President or his staff to intervene in these proceedings."

Zorn took his reassignment philopsophically, and said he plans to continue his 11-year military career whatever it may lead.

"There's been many a night when I've laid awake wondering, 'Why me?' I was at the top of the heap, but the Lord just wouldn't leave me alone."

White House press aide Ray Jenkins said yesterday he wasn't sure Carter was aware that Zorn had left his staff.