Former Environmental Protection Agency administrator Anne M. Burford testified yesterday that she may have made a "flip remark" last year about holding up a toxic-waste cleanup grant to California to avoid helping then-governor Edmund G. (Jerry) Brown Jr., but insisted that her decision was based on policy considerations rather than politics.

Her voice breaking occasionally as she brushed away tears, Burford also told the House Energy and Commerce subcommittee on oversight and investigations that she decided to be "a team player" after losing her argument that subpoenaed EPA documents should be turned over to Congress last winter.

She said the Justice Department and White House aides instead persuaded President Reagan to invoke executive privilege, touching off a confrontation that led to a contempt citation against Burford and ultimately to her resignation last March.

"I gave up," Burford said.

Much of yesterday's hearing focused on Burford's decision in July, 1982, to withhold a planned $6 million cleanup grant for the Stringfellow Acid Pits in southern California until after the November congressional elections.

A Justice Department report said an administration official later recalled Burford saying at a luncheon: "I'll be damned if I am going to let Brown take credit for that."

Under sharp questioning by Rep. James C. Slattery (D-Kan.), Burford said, "I may well have made the flip remark about Gov. Brown at that time. But there is no indication that my decision to hold back the money was anything but a good policy decision."

Brown was then a Democratic candidate for the U.S. Senate. Burford conceded yesterday that the grant "would have been a gift to whomever was governor." But, she said, "I think it would have been irresponsible for me to have made that grant at that time."

After reviewing briefing materials on her flight to California to announce the grant, Burford said, she decided to cancel a news conference on the announcement. She said that EPA officials had not approved the "Superfund" grant and that she had several unresolved questions, mainly about how much California would have to pay for the cleanup.

But Slattery said Burford's briefing packet contained indications that Superfund official William N. Hedeman Jr. had approved the grant five days before Burford's trip.

"Apparently, you unilaterally decided you were going to ignore all the agency officials who had signed off on this agreement," Slattery said.

Fifteen EPA officials, in separate closed-door interviews with the subcommittee chaired by Rep. John D. Dingell (D-Mich.), said they believed that Burford delayed the grant for political reasons.

"We were all convinced it was a political deal," said EPA's California regional counsel, Robert Thompson. "My impression was Mrs. Burford had held up because she didn't want to give Jerry Brown assistance."

His assistant, Harlan Agnew, said it was "real clear to me that there would be no . . . agreement until November or after Republican Gov. George Deukmejian was inaugurated in January."

Several EPA officials also told the panel that the dispute about California's share of the cost was resolved before Burford stopped the grant. Other EPA grants in Iowa and Florida were approved at the time despite similar questions, the panel said.

Burford also conceded yesterday that she had taken no action on conflict-of-interest allegations uncovered by her inspector general against a top aide, former EPA consultant James W. Sanderson, other than to send the information to the Justice Department.

Burford said she was "very angry" when Justice suddenly refused to continue defending her in the executive privilege case. She said some Justice officials had been "unethical."

"When we first got into this, there were repeated assurances . . . . They consistently said they would take this case all the way to the Supreme Court," she said. "Obviously, that's not what happened."