Three Middle Eastern investors signed an agreement yesterday to buy Washington's third largest banking group, Financial General Bankshares Inc., for $160 million.

If federal bank regulators approve the sale, control of the First American Banks of Virginia, Maryland and the District will be transferred to owners in Saudi Arabia, Abu Dhabi and Kuwait.

As part of yesterday's agreement, the board of directors of Financial General was expanded to include three members to represent the interests of the foreign buyers -- Clark Clifford, former secretary of Defense; Stuart Symington, retired U.S. senator from Missouri, and Gen. Elwood Quesada, former head of the Federal Aviation Administration.

Clifford's law firm has represented the Middle Eastern group in a 2-1/2-year fight to buy the Washington bank. As chairman of a Netherlands Antilles corporation that will buy Financial General, Symington will have control over the bank's stock for five years.

The Antilles company is owned by Sheik Kamal Adham, former director of Saudi Arabia's intelligence agency, Faisal Saud Fulaij, a businessman from Kuwait, and Abdullah Darwaish, manager of the financial affairs of the ruling family of Abu Dhabi.

The definitive agreement to sell the banks were announced yesterday by Financial General.

Financial General's chairman and largest stockholder, real estate man B. F. Saul Jr., was out of town yesterday and could not be reached for comment. The banks' president, former Navy secretary G. William Middendorf, was in London, his office said.

Adham and his partners agreed to pay $28.50 a share for the stock of Financial General, which has traded lately for about $22.50 on the American Stock Exchange. The price will be raised if the value of the bank increases before the transaction is completed. The Middle Eastern group already owns 18 percent of the bank's stock.

The agreement calls for Adham to make a formal offer to Financial General's shareholders as soon as state and federal banking regulators approve the takeover.

Sale of the bank will make Financial General the seventh largest foreign-owned bank in America, a federal study shows.

After a long review of foreign bank takeovers, the Federal Reserve Board recently said it had found no ill effects on the banks, the customers or the communities served by banks sold to foreign investors. The Federal Reserve said that was no reason to restrict purchases of American banks by foreigners.

Financial General's management, which had fought the takeover until two months ago, pledged to help Adham's group win regulatory approval. That could take a year or longer. The agreement gives Adham until Dec. 31, 1981 to complete the purchase, but requires his group to sell its shares if the deal does not go through by then.

Negotiations between the two sides began after a shareholder referendum in which 46 percent of Financial General's stockholders indicated they wanted to accept Adham's offer.

Both sides yesterday also signed a stipulation settling a series of lawsuits that have been going on for 2-1/2 years and have cost several million dollars in legal fees.

Adham's efforts to take over Financial General began in December 1977 when he began buyng the bank's stock with the help of Bert Lance, who had recently resigned as President Carter's budget director.

Lance bought First National Bank of Georgia from Financial General in 1975, then later sold it to Saudi financier Ghaith Pharaon. When friends of Pharaon were looking for another American bank to invest in, Lance suggested buying the whole Financial General chain.

Since that time, the takeover bid has been orchestrated by Robert Altman, a young partner in Clifford's law firm. Altman said yesterday he was "delighted" by the agreement and predicted government regulators will now approve the sale.

Sale of Financial General "will strengthen the banks and benefit the communities" by giving them greater financial resources, he said.

Altman said he could not comment on reports that Adham's group expects to attract millions of dollars in deposits from the Middle East to Financial General.

Financial General owns 12 banks with 143 offices and assets of $2.2 billion. The banks operate autonomously now, and the agreement calls for them to maintain their independence.

Recently, Financial General has begun consolidating operations of its three banks in the Washington area by changing all their names to First American Bank.

The group also includes Shenandoah Valley National Bank of Winchester, Valley National Bank of Harrisonburg, People's National Bank of Leesburg, Round Hill National Bank and Lexington National Bank in Virginia, Eastern Shore National Bank in Maryland, plus banks in Tennessee and New York.