"A working lunch with the President," the invitation said, for "only a small group of national finance council members of the Democratic National Committee."

E. A. Gregory felt reassured by the invitation, but then, last Wednesday, on the eve of the luncheon, DNC Chairman John C. White suggested to Gregory that he stay away. Outraged, Gregory called his recently acquired friend, Dr. Robert Stpaleton, the president's brother-in-law who promised to take his cause to his step-in-law, Rosalyn Carter.

Confident that White would be overruled, Gregory flew to Washington from his pensacola home last Wednesday. On his way to the White House he called White. "You'll embarrass the president," White said. And when Gregory threatened to come ahead, White warned him that he would be stopped at the gates.

So Gregory gave up and stayed away.

Who is Ed Gregory and why is the White House so mean to him?

Gregory, who is 40 years old, came from nowhere to become one of the Democrate Party's leading fund raisers.He raised and gave money to Republicans, too.

He gave $2,000 to the "Save the President" campaign for Richard Nixon and $2,000 to George Wallace in 1974. He has given thousands to gubernatorial and senatorial campaigns, among others, in Florida and Alabama. All told, he and his wife, Vonna Jo, gave $68,885 to politicans in 1976 and $20,667 last year.

Ed Gregory owns a $1 million airplane, a two-engine Beechcraft King Air 200. Miss Lillian, the president's mother has flown in it several times, he says. He often flies the president's sister, Ruth Carter Stapleton, to her religious retreat in Denton, Tex., called Holivata, to which he also has made generous contributions. One time he arranged to pick up Billy Carter's wife and three children at Plains, Ga., and fly them to Winston Salem, N.C., to attend the wedding of a Stapleton son.

On the walls of his office and stashed in boxes at his home are hundreds of photographs testifying to his moments near to power. There is Vonna Jo eating a meal on Air Force One, and there is Carter in his famous inaugural stroll up Pennsylvania Avenue with a note, "To Ed and Vonna Jo. Your help made this day possible."

But Gregory, who with his wife was indicted last month by an Alamba grand jury, is under investigation by the FBI and the Internal Revenue Service for his banking activities. With Gregory's financial affairs in chaos and his legal outlook murky, the politicans are avoiding their fair weather friend in droves.

Ed Gregory loves to tell his rags to riches life story in great detail. Like his father who died at sea during World War II, Gregory quit school at 17 at join the Navy. One day, he picked up a soldier hitchiker who took him home for the night. There he met Vonna Jo, one of nine children of a poor family, and they were married two weeks later.

In 1958, when was 20 years old, he began selling DeSotos. In a few years he had his own dealership and was investing in real estate. "I learned how to leverage out" is the way he describes how he borrowed beyond his means to pyramid his holdings.

Gregory was on vacation Pensacola in 1971 when he learned that several local propane and butane gas distributorships were for sale. He quit selling cars and bought them and merged them into a Pensacola company Energy Gas Inc. Soon the energy crisis hit, and in July, 1974, Gregory says he sold Energy Gas to Empire Gas Corp. for a $1 million profit.

In 1974, fresh from his successful gas venture, Gregory began buying motels and hotels in Florida, Alabama and Mississippi, renaming them JoVonn Inns (after his wife).

In July of that year he bought into his first bank, a tiny state bank in Thorsby, Ala. (pop. 900). "I didn't know you could bank so easy," he recalls.

Over the next 22 months, Gregory bought four more country banks, and like the motels, he renamed them all.

But from the start, the new banker was in trouble with both federal and state examiers who believed Gregory was using the banks to enrich himself, his associates and his growing number of political friends.

Through Pensacola-based Faith Investments Co. Inc., Gregory offered a full-range of services to the motels and banks - and collected hemty commissions. Faith rented them cars and sold them stationery, signs and flag-poles.

Bank officers and directors were covered by so-called "key man" life insurance, with preminums paid by the bank and commissions going to the Gregorys.

When borrowers from the banks bought credit life insurance, 47 percent of the premiums went to the Gregorys. Vonna Jo says that some months they collected as much as $20,000 in commissions involving one bank.

The examiners also criticized Gregory for allegedly overcharging the banks for his plane expenses.

But most troubking to state and federal examiners were the big loans to the Gregorys and others who lived far from the tiny communities the banks were meant to serve.

Among the loans given an "adverse classification" was $32,400 to Robert Stapleton, the president's brother-in-law.

Other borrowers included Gov. Greorge Wallace's confidante Oscar Haprer and Alabama General Bill Baxley.

Under pressure created by a series of cease-and-desist orders from banking authorities, Gregory sold off four of the five banks he controlled. Then, on Jan. 26, the Alabama Banking Department and the Federal Deposit Insurance Corp, put the last one, First Bank of Macon County, into receivership.

Since then, Gregory has sued the state and federal authorities for $10 million, alleging unlawful seizure of the bank. Another of the five banks once controlled by Gregory also has folded, and its owner has sued Gregory for allegedly mismanaging the bank.

Meanwhile, Gregory sold nearly all of his JoVonn Inns, which critics charge he "milked" in the same way he allgedly drained the banks - servicing them out of their assets.

Last month, Gregory, his wife and bank officials were indicted by a state grand jury on a technical count charging that they received deposits even after they knew the Macon County bank was insolvent.