Frustrated by their inability to win benefits for Vietnam veterans exposed to Agent Orange from either Congress or the executive branch, two veterans groups turned to the courts yesterday.

The American Legion, the nation's largest veterans organization, and the Vietnam Veterans of America, a small but vocal group, joined forces to accuse the Reagan administration of having quashed a congressionally mandated study of the herbicide's impact on the 3.1 million Americans who served in Southeast Asia.

Their lawsuits, filed in U.S. District Court here, called for a federal judge to direct the government to complete the study, which Congress in 1979 ordered the old Veterans Administration to conduct. The VA turned over the assignment to the Centers for Disease Control, which concluded in 1987 that it was unable to complete the project because of inadequate military records.

Yesterday, officials of the two veterans groups charged that the CDC, part of the Department of Health and Human Services, retreated under pressure from the White House Domestic Policy Council. They alleged that the Reagan administration was fearful of how much a positive finding would cost the government.

The groups released several memoranda from an Agent Orange working group that the Reagan White House had assembled, but conceded the memos did not contain an explicit directive that supported their thesis.

"They don't say: 'Let's kill this,' " said Mark Venuti, a lawyer with the National Veterans Legal Service Project. ". . . You have to have some conjecture, but you can see their motives."

A spokeswoman for the CDC in Atlanta declined to comment but noted that the agency recently rejected similiar charges from retired admiral Elmo R. Zumwalt Jr., a special adviser to the Department of Veterans Affairs on Agent Orange.

Veterans Affairs Secretary Edward J. Derwinski, who was named as a defendant in the suit, said in a statement, "I regret that these groups have seen fit to institute legal maneuvers that may interrupt the steady progress VA and other federal entities are making."

Derwinski won praise from the veterans' spokesmen yesterday for his rulings awarding compensation to former service personnel who suffer from non-Hodgkin's lymphoma and soft-tissue sarcomas, two types of cancer that the veterans' groups contend are linked to Agent Orange exposure. Mary Stout, president of Vietnam Veterans of America, said Derwinski needs more studies of the herbicide's impact in order to extend the benefits.

"Our nation deserves to have an answer to the question of what Agent Orange caused," said Richard Christian, an American Legion consultant. If the court approves the request, he estimated it could take two or three years to complete a study.

The veterans groups said they were unswayed by recent scientific reports that dioxin, the carcinogen in Agent Orange, may not be as potent as once feared.