Arina Ginzburg seldom listens to the post-midnight Voice of America news broadcast, but early this morning on an impulse, she tuned in.

That was how she first heard-to her utter astonishment-that her husband, Alexander Ginsburg, one of the Soviet Union's most celebrated political prisioners, was at that moment in New York, freed with four other political and religious dissidents in a dramatic prisoner exchange between the Kremlin and the White House.

An hour later, sitting tense but controlled in her Moscow apartment, Mrs. Ginzburg described both joy at her husband's release and the expectation of joining him and unquenchable sadness that others are left behind in the Gulag Archipelago of labor camps and prisons.

"My feelings at this moment are very complicated, very hard to say," she told a group of Western correspondents who had come to see her. "I am deeply happy, always happy when a person is released from prison, when someone who is sick is released, like my husband."

"But I think of Orlov [Yuri Orlov, a fellow jailed dissident] who is 53, and when Alexander Ginsburg will be cured by doctors and living with his children, I still must think about Orlov, still in prison, without his family."

As always since her husband's 1977 arrest for anti-Soviet agitation, she was dressed in a black turtleneck sweater and carrying a black woolen shawl. She had thought often of an exchange, she said, but the uncertainly of not knowing "whether it would be or wouldn't be" was so painful she "made a decision not to think about it."

Such a gesture is in keeping with a woman whose life with her husband has been dominated by prison and by dissent. Indeed, Mrs. Ginzburg, 41, was married to her 42-year-old husband in a labor camp-a rarity in the Soviet prison system achieved only after Alexander Ginzburg and some friends went on a hunger strike in 1969 to force the prison authorities to allow the ceremony. The bride carried a bouquet of yellow flowers gathered from a fence around the camp.

The swiftness and secrecy of the exchange left her grasping to fit it into her own life, which is deeply enmeshed in dissident political activities. She recalled that yesterday afternoon she sent an open letter asking visiting French President Valery Giscard d'Estaing to intervene on behalf of imprisoned dissidents, including her husband. "He was already in New York," she suddenly realized.

She last saw him March 23 during a three-hour visit to Labor Camp No. 17, in Mordovia, several hundred miles southeast of Moscow. With prison guards present, they were allowed to talk to each other only across a long bare room and only about family matters. She expected to see him next in September.

Mrs. Ginzburg took over her husband's work as director of a relief fund for political prisoners and their families after he was arrested, then convicted last year of anti-Soviet slander and agitation and sentenced to nine years' labor. The fund is supported by royalties from books by Alexander Solzhenitsyn, the exiled Nobel laureate novelist who is a close friend of Ginzburg's.

The exchange also includes the immediate family of the freed men, but Ginzburg never sought to leave the Soviet Union, and actively resisted such pressures during his life of challenging the state. He has been imprisoned three times for political reasons since 1960.

Immediate family includes Ginzburg's 72-year-old mother, who has been in bed for several months at home with heart trouble. Arina Ginzburg said she had not yet had time to talk with her mother-in-law to discuss their plans. She said she has heard nothing either from the U. S. Embassy, which declined any comment, or from Soviet authorities.

The couple has two children, Alexander, 6, and Alexei, 4. When her husband was arrested two years ago, Mrs. Ginzburg told the little boys that he was sick and being treated in a hospital. She has sent them toys in his name from time to time to continue the fiction.

"Now, I'll just tell them the hospital is in New York," she said, with a rare smile, "and we'll be joining him . . ." CAPTION: Picture, ALEXANDER GINZBURG . . . one of dissidents released