The Federal Aviation Administration is negotiating with the bankrupt Braniff International Corp. and PSA, an expansionist California airline, to seek an out-of-court agreement on whether the takeoff and landing rights that Braniff once used are property and, if so, who owns them.

At stake are the operating rights of several airlines at many airports, jobs for as many as 2,000 of Braniff's 8,000 former employes, and the amount of repayment Braniff's creditors will receive.

When Braniff grounded its fleet last May and sought protection of the federal bankruptcy court, it held 411 of the rights, or slots, nationwide. About 350 of those slots have since been reallocated by lottery to other airlines; the FAA has held about 50 at Dallas-Fort Worth Airport because of air traffic control capacity problems following the controllers' strike of August 1981.

PSA is seeking 99 slots at Dallas-Fort Worth alone as the hub of a "Texas Division" that would use former Braniff equipment and many Braniff employes and would interconnect with PSA's Pacific Coast network at six western cities.

If that ambitious plan succeeds, the PSA symbol would be flying coast to coast, with stops at Washington National and 15 other airports.

As part of a complex ruling, a Fort Worth federal bankruptcy judge, John Flowers, ordered the FAA on Jan. 31 to give the slots to PSA.

Flowers said that, if Braniff simply were liquidated, its unsecured creditors could expect only 9 cents on the dollar, while an agreement with PSA to operate the Texas Division would enable the creditors eventually to receive at least 12 cents on the dollar.

The Justice Department, on behalf of the FAA, and several airlines appealed, and those concerned are awaiting a ruling from U.S. District Judge Eldon Mahon in Fort Worth that could come as early as next week. Most attorneys active in the case think it is certain to reach the Court of Appeals, and perhaps the Supreme Court.

"We would rather work this out if we can, and the judge would rather we did, too," said J.E. Murdock III, FAA's chief counsel.

Howard D. Putnam, chairman and president of Braniff Airways Inc., said Thursday he was more optimistic that an out-of-court settlement could be reached than he had been earlier.

Putnam said the problem is that, "With no slots, you have no deal with PSA . What has frustrated me is that we are trying to put 2,000 people back to work, which seems to be a priority of the Reagan administration, and we're being shot in the foot by the administration . . . ."

An out-of-court settlement that would make everyone happy will not be easy to achieve. The Texas Division is planning flights into cities such as Houston, where Continental Airlines has been seeking additional slots, and Chicago, where United wants more room. Thus, if the FAA finds slots for the Texas Division in those two cities, questions arise as to what will be done to satisfy earlier requests.

Further, many airlines now are using what once were Braniff's slots. Are they going to have to give them back? "Not without a fight," an executive from one of those airlines said. He declined to comment for attribution because the case is in court.

The FAA's legal position is that the PSA--Braniff proposal for a Texas Division does not represent a successor company to Braniff. FAA had agreed that it would provide slots to a legitimate successor.

According to sources familiar with the case, FAA's longer-term concern is that, if slots come to be regarded as property, they would have to be sold, thus creating a rulemaking and adjudicatory nightmare, with the FAA a potentially liable party in endless airline fights over airport rights.

"It would be a fairly drastic change to the regulatory system under which we deal," said Murdock.

Since the air traffic controllers strike, the FAA has been forced to allocate slots, sometimes through lottery, and slots are undeniably valuable, just like property. The long-term resolution of how slots will be allocated among deregulated airlines operating in a system that has air traffic constraints must be decided sooner or later, either in Congress or by the courts.

It's not just the airlines that are concerned. The general aviation community--both the business flyer and the private pilot--have a major stake in airport access being generally unrestricted. If slots become property, would corporate jets have to buy them from airlines for access to major airports?

FAA Administrator J. Lynn Helms, in a letter to Putnam that predated the bankruptcy ruling, pointed to another factor when he said that giving the Braniff slots to the Texas Division "would seriously disrupt the FAA's statutory duty to manage and maintain a safe and efficient airspace system."