Kathleen Kelly O'Day, 64, a mathematics teacher at Wheaton High School from about 1960 until her retirement in 1974, died Saturday after being stricken by a heart attack at her home in Kensington. She had been suffering from a lung ailment for some time.

Mrs. O'Day was born in Yonkers, N.Y. She graduated from Mount St. Vincent's College in Riverdale, N.Y., and then taught in Yonkers before moving to the Washington area in 1952. She earned a master's degree at the University of Maryland and began her teaching career in Montgomery County about 1954. She taught at parochial and elementary schools in the county before moving to Wheaton High.

She was a member of the Maryland Teachers Association, the National Education Association, the Mathematics Teachers Association and the Montgomery County Teachers Association.

Survivors include her husband, Francis D. O'Day, of the home in Kensington; two daughters, Margaret O'Day Reilly, of Gaithersbury, and Katleen O'Day O'Brien, of Lexington Park, Md.; six sons, Thomas F., of Irvine, Calif., Michael S., of Houston, Texas, D. Kelly, of Philadelphia, Sean P., of Wheaton, F. Brian , of Portland, Ore., and Peter J., also of Lexington Park; a sister, Mary Kelly Power, of Scarsdale, and 11 grandchildren.

The family suggests that expressions of sympathy be in the form of contributions to the Ladies' Auxiliary of the Kensington Volunteer Fire Deparment, Kensington, Md.