Grocery stores would be required to offer consumers rain checks or substitute merchandise when they run out of advertised products, or at least warn that only limited quantities are available, unless they can document that they had anticipated stock enough to meet reasonable demand, under a Federal Trade Commission staff proposal released yesterday.

The commission is scheduled to meet Monday to consider the proposal.

The proposal is designed to end more than a year of controversy touched off when the staff first recommended modifying or eliminating the current rule that requires grocery stores to stock the items they advertise.

After a July 1984 recommendation, consumer groups charged that the staff wanted to kill the rule. Consumers Union said yesterday that the latest proposal is a retreat from last year's staff report, which said the rule "does not appear to be justified."

Congressional FTC watchers had predicted the changes as a way for the FTC to save face after receiving an avalanche of protest to the staff's original proposal.

The FTC received about 3,600 letters in response to the original proposal, including a letter from White House consumer affairs adviser Virginia H. Knauer, who said "the rule is functioning well." Knauer recommended allowing rain checks or substitutes for out-of-stock items.

The new staff proposal would leave the existing requirement intact, but the changes would provide new ways of complying with the 1971 rule, which applies only to food stores. Currently, a grocery store that runs out of an advertised item is subject to fines of up to $10,000 unless it can document that it stocked enough of the item to meet "reasonably anticipated demand." Offering a rain check or substitute, or warning of limited quantities, is not a defense under the existing rule.

Consumers Union said the new staff proposal is a "substantial improvement" over the idea of scrapping the rule. But the new proposal does not go far enough in clarifying the rule for business or assuring good-faith compliance, said Mark Silbergeld, director of the group's Washington office.

The Food and Marketing Institute, which has urged the commission to throw out the rule, said yesterday that it had not yet evaluated the new staff recommendation, but may be open to compromise.

"We don't think there is any need for this rule, but it is certainly possible that amendments could eliminate the unnecessary costs without outright repeal," said spokesman George Green.

In 1981, there were about 165,000 grocery stores in the country with total sales of more than $240 billion, the FTC said.

Both the old and the new staff reports argue that the existing rule imposes costs upon grocers far outweighing the benefits to consumers. To comply with the rule, grocers may stock more goods than they would otherwise, spend more on documentation and incur extra legal costs, the new report says. These costs are passed along to consumers in the form of higher prices and may exceed benefits by $57 million to $173 million a year, the report says.

The staff argues that the new proposed changes would reduce these costs and put food stores on an equal footing with other retailers, which are not subject to punishment if they run out of advertised items that had been stocked to meet "reasonable demand." All retailers are subject to FTC rules against deceptive advertising and "bait" advertising.

"The rule may provide an illusion of protection," the new staff report said.

Consumers Union has argued that other modifications would simplify compliance and reduce grocers' costs. Silbergeld also said that any warning of "limited quantities available" should specify the amount available of any special sale item.

The FTC has taken action against 10 alleged violators of the rule in 15 years, and only one case went to trial. In that case, a grocer who had run out of an item and offered rain checks was found to be in violation of the rule, the FTC said.