Charles B. Ferster, 58, a psychology professor at American University whose research in experimental psychology helped lay the groundwork for the psychotherapeutic technique known as behavior modification, died of a heart attack Tuesday at Washington Hospital Center.

Dr. Ferster was the coauthor, with psychologist B. F. Skinner, of "Schedules of Reinforcement," a leading textbook in the field of experimental psychology, and the author of more than 80 scientific papers and articles.

In the late 1950s, he established a major publication in the field of experimental research, the Journal of the Experimental Analysis of Behavior, while he was an assistant professor of psychology at the Indiana University School of Medicine.

It was at Indiana University that Dr. Ferster, a former research associate at the Yerkes Laboratory of Primate Biology in Florida, began preliminary research on an experiment in language communication between man and chimpanzees that he believed would reveal new knowledge about man's verbal behavior. This knowledge would, it was hoped, lead to new techniques in teaching the very young or emotionally disturbed children to read.

As a result of this work, he began treating children afflicted with infantile autism with behavioral modification techniques.

After moving to the Washington area in 1960, Dr. Ferster continued his work with primates and the study of learning in humans and animals at the Institute for Behavioral Research in Silver Spring, a nonprofit research and educational organization he helped found.

He served as the institute's executive director and as a senior research associate until 1967, when he joined Georgetown University's psychology department. In 1969, he became a professor in the psychology department at American University, and had served as chairman of the department for three years.

Dr. Ferster, whose work at American University included applying rigorous behavioral approaches to human problems, wrote several papers on the functional analysis of clinical depression, and was credited with initiating at the university what has been called a revolutionary experiment in undergraduate education: the Learning-Center Approach, or self-paced learning.

He also worked as a volunteer therapist at the D.C. Institute of Mental Hygiene, which serves Washington's needy, and was director of research and training at Chestnut Lodge Sanitarium in Rockville, a private psychiatric hospital.

Dr. Ferster, who lived in Washington, was born in Freehold, N.J. He graduated from Rutgers University and earned master's and doctoral degrees from Columbia University, where he also lectured. He was a research fellow and instructor at Harvard University. He served in the Army Air Force in World War II.

He was a fellow of the American Psychological Association and the Psychonomic Society.

Survivors include his wife, Elyce, and four children, Andrea, Samuel, Warren and William, all of Washington, and a brother, Paul, of Woodbride, Conn.

The family suggests that expressions of sympathy be in the form of contributions to the D.C. Institute of Mental Hygiene.