The Carter administration has told United Airlines and the International Association of Machinists that their new contract, ratified after a 58-day strike, is in "probable noncompliance" with anti-inflation guidelines.

The action, which was reported to both sides on Tuesday and confirmed by company executives yesterday, sets the stage for a new test of the administration's embattled guidelines program.

The United contract is technically exempt from the administration's 7 percent standard for annual wage and benefit increases so long as it fits a "pattern" set by Trans World Airlines last fall, before the guidelines were announced.

United President Richard Ferris contends that the contract, which he has described that the contract, which he has described as costing 34 percent over three years, conforms to the TWA pattern and thus complies with the guidelines. The Council on Wage and Price Stability refused to comment on the issue yesterday but apparently had serious enough doubts to begin proceedings against United, which, like other airlines, is also plagued by the grounding of its DC10 jumbo jets.

The administration's notice to United indicates a toughness that was not apparent two months ago when, without hesitation, it sanctioned a Teamsters contract that was described by both industry and unuon negotiators as exceeding 30 percent over three years. Its handling of the United case - and the airlines' response - will undoubtedly be watched closely by other companies and unions with negotiations this year.

While the "notice of probable non-compliance" that is being sent to the company and the union does not mean that the contract has been rejected by the government, "it says the information available so far suggests they are not in compliance and they have some more explaining to do," said one government source.

The notice is the first step in a three-stage process that could lead to the public branding of United, the nation's largest domestic air carrier, as a violator of the anti-inflation program - an action the government has yet to take against any company.

Outside of public condemnation, it is uncertain what the government could to to a violator, however.

U.S. District Judge Barrington D. Parker ruled last week that President Carter exceeded his constitutional authority in threatening to deny federal contracts to guideline violators because such economic sanctions had not been authorized by Congress.

The government's appeal of Parker's ruling will be heard by the U.S. Court of Appeals next Wednesday. Because Parker's order has been stayed until the appeals court acts, the government is techinically free to continue threatening contract debarment. But officials have indicated this will not be done so long as a final resolution of the issue appears imminent.

While United hauls tons of U.S. mail and thousands of traveling bureaucrats, some officials privately question whether the disputed sanctions would be effective against United anyway - especially after militant machinists twice rejected less expensive contracts during a strike that cost the airlines an estimated $80 to $90 million.

A company spokesman said a check of United contracts shows none in excess of $5 million, which is the threshhold for the debarment sanction.

Neither United nor the IAM indicated how they will respond to the government's notice. United said it will review the notice and reply by June 22, as required by the CWPS review process. An IAM spokesman said the union is "considering its options." Neither side is considered likely to be willing to reopen the contract.

A CWPS oficial said United is not the first company to be challenged on wages but declined to identify the others. Two companies, amarada Hess Corp. and the cement division of Ideal Basic Industires Inc., have been cited for noncompliance with the price standard but have not exhausted appeals procedures.

There was some speculation in administration circles that guideline rules might be relaxed for a company that has taken a long strike to hold down labor costs. An administration official said CWPS notice to United appears to indicate a reluctance to bend the guidelines any further. "You're either in or you're out," he said.