A federal judge yesterday ordered Hustler magazine to pay $431,879 for unauthorized publication of 11 nude photographs of Elizabeth Ray, who said she was hired as a Capitol Hill secretary solely to be the mistress of former Ohio representative Wayne L. Hays.

The award, plus attorneys' fees to be determined later, was made here to Barry M. Blackman, a New York City photographer whose pictures of Ray appeared in Hustler in September 1976, shortly after Ray made her accusations.

The magazine reprinted some of the photos several years later.

U.S. District Court Judge Joyce Hens Green, who entered the award yesterday, had ruled after a trial last year that Hustler was liable for copyright infringement. She said the award "will amply compensate plaintiff for his losses . . . strip defendant of its ill-gotten gains, and discourage similar actions" in the future,

Last year Green found that Hustler publisher Larry Flynt had acted with "deliberate, intentional, malicious, and wanton" disregard of Blackman's rights by publishing the photos after Blackman had rejected his offer to buy them.

When the photos were taken in 1972, Ray signed a release giving Blackman all rights to publish and copyright them. She receives no money from the award.

Hays denied Ray's charges in early 1976, but resigned later that year.

Norman Zafman of Los Angeles, an attorney for Hustler, said yesterday he could not comment on Green's 24-page ruling because he had not read it, but added, "When the judge goes against you, it's always too high."