The Fairfax County School Board, resisting pressure from mainly conservative and religious forces, voted 7 to 2 last night to study the possible expansion of the county's sex education program.

In so doing, board members stressed that approval of the study proposal did not necessarily mean that they feel the county should lift its ban on teaching students about homosexuality, abortion, masturbation, rape and incest.

"Unless we have something before us that we can discuss, it becomes a useless debate," said board Chairman Rodney Page. He said the proposal, which calls on the school staff to study alternatives ranging from elimination of the county's controversial sex education course to a broad expansion, is merely intended as an agenda for discussion.

No changes are set to take place in the county curriculum until the fall of 1981.

The board's decision followed more than an hour of stormy comment by county residents, in which most speakers criticized school sex education courses as an immoral exercise that promotes teen-age promiscuity and invades the privacy of both parents and students.

"In the family life curriculum of Fairfax County schools is to truly assist the family and churches, then it is the family and the church which should be telling the schools what assistance they may need," said the Rev. Thomas J. Welsh, Roman Catholic bishop of Arlington in a statement read to the board by a spokesman. "The school can play a helpful role in this area, but only when it sees that role as secondary and auxiliary to the family and the church."

The antisex education lobby drew sustained applause and cries of "amen" from an enthusiastic crowd of 200, as 10 speakers in all assailed the continuation of a course of instruction one called an "evil, satanic program presented . . . in a purely brothel atmosphere."

The sex education study proposal was brought to the board by the school system's instruction and curriculum committee after a recent school survey indicated that 70 percent of parents were dissatisfied with the present program and about 90 percent favored a more liberal one.

Despite what appeared to be an overwhelming majority among parents of Fairfax students, proponents of a more liberal sex education policy were far outnumbered at yesterday's meeting.

Only two addressed the group, and they were heckled by the crowd.

"We can no more hide sex from our young people than we can put a fig leaf around the wind," said the Rev. Alvin J. Horton. "It is time that we in Fairfax County stop playing with sex education as though it were a mere extracurricular pastime and begin a serious investigation into how we, as responsible adults, can help our young people confront the many and varied facets of human sexuality . . ."

Voting against the study proposed were board members Gary Jones and Robert E. Smith, who also launched an unsuccessful attempt to put the board on record as favoring a permanent ban on discussion of homosexuality and abortion in county schools.

Jones warned his fellow board members so tread softly on the topic that, he said, amounted to an intrusion of the values of the family and the individual.

"We can certainly be taken into court on this because a taxpayer would be putting public funds into teaching a course which is the anthesis of their beliefs," he said.