Frederick Arell Marsteller, 63, the president of Marsteller, McCabe & Co. Inc., a real estate firm which he helped found, died of cancer Monday at Alexandria Hospital.

Mr. Marsteller went into the real estate business in the Washington area in 1946. The firm that bears his name was formed about 1950 and is active in the District, Maryland and Virginia.

In the course of his career, Mr. Marsteller held many posts in real estate industry organizations. He was president of the Washington Board of Realtors in 1969 and had been a director and regional vice president of the National Association of Realtors. He also was a director of the Virginia Association of Realtors, the Metropolitan Board of Trade and the Columbia Real Estate Title Insurance Co. He taught real estate courses at the American University business school.

Other professional organizations to which he belonged included the Northern Virginia Board of Realtors, the Montgomery County Board of Realtors, the Maryland Association of realtors, the Virginia Association of Realtors and Omega Tau Rho, an honorary real estate organization.

Mr. Marsteller was the son of a Navy physician, Capt. A. A. Marsteller, and was born in Havana while the family was stationed in Cuba. He grew up in the Phillippines, China and Washington, where he graduated from the old Western High School.

He attended the Georgetown University School of Foreign Service and graduated from Washington and Lee University.

During World War II, he was an officer in the Navy and saw service in the South Atlantic and the Southwest Pacific, where he commanded a submarine chaser in the New Guinea campaign. He was a lieutenant commander at the time he left the service to begin his real estate career.

Mr. Marsteller was a member of the parish of St. Mary's Catholic Church in Alexandria and of the Parish Advisory Council. He also was a member of the St. Mary's Housing Corp.

Survivors include his wife, the former Ruth Quinn of Alexandria, where the family lives; four daughters, Karen Nash of Newport, R.I., Clair Tieder of Oakton, Marilyn Marsteller of Cincinnati, and Margot Marsteller of Alexandria; two sons; Frederick A. Jr., of Middletown, Conn., and David A., of Alexandria; a brother, William A., of Centerville, Ohio, and four grandchildren.

The family suggests that expressions of sympathy be in the form of contributions to the American Cancer Society, or to the Realtors Memorial Fund of the Northern Virginia Board of Realtors.