A group led by Mesa Petroleum Co. Chairman T. Boone Pickens Jr. won a key court battle yesterday in its effort to take over Phillips Petroleum Co.

A Delaware chancery court judge ruled that an agreement by Mesa last year to refrain from buying stock in General American Oil Co. for five years could not be extended to cover Phillips, which has since bought General American.

Vice Chancellor Joseph Walsh ruled that Phillips cannot press an Oklahoma case that has barred the Pickens group from purchasing stock in Phillips, the nation's ninth-largest oil company.

Phillips said it would appeal Walsh's decision to the Delaware Supreme Court. Although its headquarters is in Bartlesville, Okla., Phillips is incorporated in Delaware.

The company added it would continue other court challenges to Pickens' group, which calls itself Mesa Partners and includes affiliates of Mesa, Pickens and outside investors. The group, which already owns 5.7 percent of Phillips, is offering $60 a share for 15 percent more as the next step in a planned $9.2 billion takeover of the company.

The point of law on which Phillips last week won a restraining order in Oklahoma against the Mesa bid is the same on which the Delaware court ruled yesterday. Mesa Petroleum had obtained a large block of General American, and then sold it back to the company, which won the "standstill" provision that prevented Mesa from buying more GAO stock.

Phillips had argued that the standstill agreement also applied to purchases of its stock, even though the agreement didn't specifically mention Phillips. The company said Pickens knew Phillips planned to buy GAO when he signed the agreement. Phillips purchased GAO the day after the agreement was signed in January 1983.

"Even if all attendant circumstances are considered, Phillips' claim that the standstill agreement contemplated its inclusion as a future acquisition target is unreasonable and beyond the scope of the discernable intention of the parties," Walsh wrote in his decision.

Mesa Partners had put its offer on hold pending resolution of the Delaware case and other legal disputes. A group of Phillips wholesalers yesterday won an order in a separate Oklahoma case blocking Mesa Partners from buying Phillips stock, and Louisiana officials also have won an injunction against Mesa.