Frank McCallister, Naval Academy offensive lineman who earlier this week was selected to the Associated Press third all-American team, was ruled ineligible yesterday for the Garden State Bowl.

Navy will play Houston Dec. 14 in the bowl game at East Rutherford, N.J.

McCallister, a senior from Pittsburgh, played one year at the University of North Carolina. He transferred to the Naval Academy and has played on the varsity the last three years. He played on Navy's junior varsity in 1977.

The Eastern College Athletic Conference has granted national service academies a waiver allowing transfer students to compete for five years, if the student plays on a subvarsity level the season after he transfers.

But the NCAA conducts postseason play and does not grant such a waiver. So, under its rules, the NCAA ruled McCallister ineligible.

McCallister, a 23-year-old 250-pounder, started five games at guard and six at tackle this season. A Navy spokesman said McCallister was disappointed by the ruling and preferred not to talk about it.

"We knew about this (possibility) the first day he came here," said the spokesman. "But when he came here, we weren't a good team. Who would have told you four years ago that we would be a bowl team in 1980? It's a well-documented NCAA rule and there's nothing we can do about it. But it's a shame for Frank."