A presidential cost-cutting commission acted illegally when it endorsed three recommendations to trim expenses in federal food programs, a federal judge has ruled.

In a ruling sought by a coalition of advocacy groups, U.S. District Court Judge Gerhard A. Gesell found that the President's Private-Sector Survey on Cost Control went out of bounds when it urged President Reagan to redefine family size for food-stamp purposes, and to count school lunch benefits against food-stamp eligibility and then eliminate the $10 monthly minimum benefit.

Such recommendations, he said, involve "substantive legislative policy issues" and cannot be approved except by a committee " 'fairly balanced' to represent the points of view affected."

"While it is of course true that cost savings can always be accomplished readily by repealing legislation that grants specific benefits, recommendations designed to accomplish such repeal do not fall within the narrow area of cost and management control . . . ," Gesell said.

The ruling was a partial reversal of an earlier decision by Gesell, who had dismissed the case in February on grounds that there was no evidence that the panel intended to enter the policy arena. Weeks later, however, the panel, headed by industrialist W. Peter Grace, released a report recommending changes in food-stamp policy.

This time, while Gesell again dismissed the coalition's complaint that the survey's deliberative process was a "sham," he upheld its complaint about the food-stamp recommendations.

The Food Research and Action Center, one of the plaintiffs, said the ruling "casts serious doubt on other recommendations made by the Grace committee." The panel has conceded that many of its cost-cutting proposals would involve changes in the law.