The traditional 40-hour workweek will be replaced by a shorter, more flexible work schedule in the 1990s, according to a new report by a national labor studies group.

Rapid growth in the number of working mothers, increased energy costs of commuting from home to work, and workers' demands for larger blocks of personal time are a few of the "unprecedented changes in society and in the composition of the workforce" that "may . . . result in the end of the traditional 9-to-5 day and the 40-hour week," according to the report by Work In America Institute, Inc.

Entitled "New Work Schedules for a Changing Society," the report, based on an 18-month study by the institute, comes when the demand for more leisure time is conflicting with employers' determination to increase production and hold down labor costs.

The government itself, in its dealings with the Professional Air Traffic Controllers Organization, illustrated continued employer resistance to calls for reduced hourly work schedules.

The controllers said they deserved a shorter workweek, by as much as eight hours, because their jobs were stressful. The government said the proposed cut in working hours and the controllers' concurrent demand for more pay were too expensive. Nearly 13,000 controllers walked out in protest Aug. 3. Only about 1,000 of those heeded President Reagan's order to return Aug. 5. The rest were fired.

Despite that development, the institute, based in Scarsdale, N.Y., predicts that the basic workweek for nonagricultural workers will be reduced to 36 hours by 1990 "with new options for the nine-hour day, four-day week as a pattern to replace the standard workweek."

The institute also believes that by the next decade, 25 percent of fulltime, nonagricultural workers in the United States will be on flexible work schedules, compared to 11.9 percent working on such schedules today. "Flexitime" means workers choose their starting and quitting times within management-set limits.

Part-time work and job sharing will continue to grow, affecting 28 percent -- compared to 21 percent now -- f the nation's total nonagricultural workforce, according to the institute's report.

The 1990s "may be the decade in which Americans free themselves from the tyranny of the time clock," said institute president Jerome R. Rosow. He said business necessity, as much as employes' needs and wants, will bring about the changes.

"New work schedules in the United States, after a slow start in the 1970s, have been expanding rapidly," Rosow said in summary of the institute's report.

The 128-page report argues that flexible work schedules yield higher productivity "reflected by fewer paid absences, less idle time . . . less overtime pay, and better use of production facilities." Employe morale also improved, the report said.

However, the report said "the single most important obstacle to new work schedules is still the autocratic tradition of supervision, founded on the belief, deeply embedded by custom and practice, that rigid work schedules are essential to efficiency."