Yale University is turning over its football and lesser athletic fortunes to a mathematician, following the lead of the Cleveland Browns 15 years ago.

When Frank Ryan was at the zenith of his powers as a thinking man's quarterback, an undereducated press would never let the public forget that he had a doctorate in geometric function theory.

Indulgently, while awed scriveners took care with the spelling, Ryan one more time would go over the title of the thesis - "A characterization of the set of asymtotic values of a function holomorphic in the unit disc."

Nobody at Yale is expected to blink at the spectacle of athletic director Ryan also lecturing in mathematics, which he is scheduled to do on the side.

Teaching will not be new to Ryan. He took his doctorate while still playing pro football and also was an associate professor of mathematics at Case Western Reserve University, posing the question of which field was his avocation.

It took a locker room sense of humor to survive the sticky situation created by a well-intentioned but runaway imagination in the Cleveland Browns' public relations department.

Because as an undergraduate at Rice University he received a baccalaureate degree in physics, the Browns' press brochure reported that he had an "IQ in the genius class." It inspired a story that he wanted to become a nuclear phsyicist.

He explained to little subsequent avail that, "I was taking physics as an undergraduate - atomic, nuclear, solidstate, mechanical, thermodynamic and electronic.

"But I never had any notion of being a nuclear physicist as such . . . I had no illusions about being a great intellect when I looked around me in my classes. That's why I was ambarrassed by these 'genius' references. My classmates must have wondered about me."

If Ryan at 40 is prematurely gray ("from too may third-downs-and-long-yardage," he says and suitably donnish for the Ivy League, he also can be puckish.

He went to Cleveland as a veteran, after four years with the Los Angeles Ramas, who drafted him on the fifth round. As a quarterback-leader he set the tone for horseplay.

He used to call rookies on the telephone and pose as a sports writer, asking the most embarrassing questions.

A news photographer who once dared to needle him regretted it later when he developed a picture of Ryan that was a scoop of sorts because it showed him to be lefthanded.

Ryan, as a backup quarterback to Sonny Jurgensen in Washington, saved practice from being terminally dull for reporters with his bantering while holding the first-down sticks for scrimmages at about $70,000 a year.

Ryan got in that fancy neighborhood with the Browns by perpetrating an upset of Johnny Unitas and the "Baltimore Colts for the National Football League championship in 1964, a victory that rivaled that by the New York Jets over the Colts in a Super Bowl game.Cleveland's triumph was more decisive, 27-0 and Ryan threw three touchdown passes.

He was selected for the Pro Bowl three times during his career.

Ryan is paying retroactively.He recently underwent surgery for a ruptured cervical disc, which involved grafting a piece of hipbone onto his neck.

Under pressure from an unruly and beefly linebacker named Myron Pottios, then with the Rams, in a 1968 game, he ducked when he probably should not have, his helmet ended a couple of feet deep in Potios' belly and, when the quarterback, recoiled, he had a jumped neck.

The injury took on the symptoms of a pinched nerve while Ryan was going through his last two coaches, Lombardi and Bill Austia. Occasionally, he suffered numbness in one arm.

With the Rams, his sternum (chestbone) was separated from his collarbone. One Pro Bowl game resulted in a separated shoulder. At Cleveland, he went into a playoff game with both shoulders, both knees and both ankles banged up, as if he wanted to keep his condition symmetrical.

Yale can savor the knowledge that Harvard once talked to Ryan about coaching its football team.

Ryan made a kind of history when he was invited as director to information systems for the House of Representatives to computerize the antiquated voting system.

He is a native of Fort Worth, a Texan who walks tall at 6-foot-3, thinks big but doesn't talk big, except to say self-mockingly that, "My goal is to beat Harvard."

He also said, "Winning is more fun than losing, but not winning at all costs. I don't think athletics are the end-all of one's life and career."

Behind those cool eyes is temper.

Vince Lombardi, who also fancied computers, once said his at Green Bay were programmed to tell everything about a player. "I don't like players who score too high intellectually," the coach remarked, roguishly. "They don't tend to hit."

When that was relayed to Ryan, he reacted reflexively with the wounded ego of the typical athlete. "I hit!" he exclaimed with a sudden steely glint in his eyes. Almost as quickly, as if he realized he has lost his customary cool, he added, a bit sheepishly - "mentally."