More than 60 percent of the nation's 17-year-olds spend less than five hours a week on homework, and most watch at least one hour of television on school nights, a new study shows.

The more homework the teen-agers did and the less television they watched, the better they fared on a nationwide math test, the National Assessment of Educational Progress reported yesterday.

The study was based on a questionnaire given 10,000 students who took the test in 1976. The National Assessment, a research group, is funded by the federal government but run under contract with the Education Commission of the States, which is based in Denver.

Thirty percent of the 17-years-olds said they do 5 to 10 hours of homework weekly, and 6 percent said they spend more than 10 hours. Slightly more than half reported doing less than five hours, while an additional 7 percent said they didn't do their assigned homework, and 6 percent said none was assigned.

Nearly half said they watched no television or less than one hour on school nights (Sunday through Thursday), but 35 percent watched from one to three hours nightly, 12 percent watched from one to three hours nightly, 12 percent watched three to five hours nightly, and 5 percent spent five hours or more in front of the tube nightly.

The students who had no assigned homework did poorly regardless of how much or how little television they watched, the study said.

"These students answered only an average of 50 to 55 percent of the questions correctly regardless of whether they watch an hour or up to five hours of TV nightly," it said.

At the other end of the scale, the high scorers - 80 percent correct or higher - "watch less than an hour of television at night and do more than 10 hours of homework in a week.

"Those 17-year-olds who do between five and 10 hours of homework a week and watch less than one hour of TV nightly answered 73 percent of the questions correctly," the study added.

But it said mixing a fair amount of homework with a lot of television apparently doesn't improve grades.