The governor's staffers call it "the ogre"--the foot-tall metal file box that is the official receptacle for letters and telephone calls of invitation for Gov. Harry Hughes.

Each Friday afternoon a state trooper lugs the box across the street to the governor's mansion, where only Patricia Hughes will help her husband select and discard, deciding whom the governor shall grace with his presence and whom he shall not.

The selection is a sensitive political task and it has become a subject of note in Annapolis. For the first time in memory, no aides or political advisers are involved in the process. That Pat Hughes thought up the box tells much about the control she exerts as Maryland's First Lady.

She is the governor's wife, a post for which there is no apprenticeship, one that is undefined and constantly scrutinized for flaws. Her husband calls her an "unofficial adviser," but she is widely seen as more influential than that.

Pat Hughes' behind-the-scenes involvement, which she insistently downplays, extended to her husband's 1982 reelection campaign, his staff-reshuffling early in his first term, and the administration's high profile support for women's issues and Medicaid funding of abortion.

She was also seen as one of the causes of the unpleasant fallout between Hughes and his first lieutenant governor, Samuel Bogley, who with his wife Rita is an ardent right-to-life advocate.

Pat Hughes' role has prompted the sort of criticism normally reserved for the more publicly powerful: She has been called "icy" by Baltimore's mayor; paranoid by State House denizens; imperious, inflexible and humorless by Hughes' staffers, and snooty by some legislators, who proclaim a "regal" attitude at Government House, the official name of the mansion and the one by which Pat Hughes would have it called.

For the most part, critics will speak about her only with the sworn promise that their names not be used. They say she never forgets a slight and willingly extends her memory to her husband, whom she accompanies to nearly every political and social event.

Her defenders say Pat Hughes is much misunderstood. They say she is a disciplined, extremely precise and intelligent woman--as one would expect from a former teacher, a 53-year-old Bryn Mawr graduate who was an undergraduate at a time when dressing for dinner was still required.

And, having grown up on the Eastern Shore, where she met her husband more than 30 years ago, Pat Hughes is a reserved woman who has difficulty relaxing except among the closest of friends. She is also blunt, often appearing unable or unwilling to massage egos.

They say these qualities, combined with her role as preserver of the privacy of a laid-back governor, are threatening in the male-dominated world of politics.

Pat Hughes is not especially well liked in Annapolis, but it is hard to know whether any governor's wife could be. It is a difficult position, and the women who have occupied it have reacted to it in different ways.

Former acting governor Blair Lee III's wife Mimi disliked it so much she spent only half her time in Annapolis, living mostly in the couple's Silver Spring home. Former governor Marvin Mandel's first wife, Bootsie, appeared to like it so much that she refused to vacate the premises when he announced their separation, causing a national uproar.

His second and current wife Jeanne loved it--and the power of the governorship--as much as any of them have. When Mandel was hospitalized at one point, she literally took over his office and ran the staff.

As one person put it: "There is no winning in that job. I might want to be governor but never the governor's wife. It's a horrible burden."

Said Pat Hughes: "This is not a place full of very secure people, that again goes with the political ego." She added, "The pay's not very good either." It is one of the few jokes she allows herself during a 90-minute recent interview.

Pat Hughes is not particularly forthright on her involvement in her husband's government. "I was not elected," she said. "I don't get involved. If I feel strongly about something I ought to run for office."

But consider:

After his first year in office, Hughes, suffering from bad press and bad relations with the legislature, reshuffled his staff in a major way, removing a chief of staff and shifting others around. Of his wife's role in this Hughes said, "The changes weren't originated by her, but she pushed it." She said, "I realized it was extremely important that a competent person be in that chief of staff position."

One former Hughes staffer, who left in part because of Pat Hughes, said, "you have to get along with her to get along with him. She can ruin your career and there are enough corpses around to prove it."

During the 1982 reelection effort, Pat Hughes was a key member of the weekly strategy meetings and it was she, more than anyone, who could scratch a suggestion by saying, "that's not Harry."

Said one regular at the strategy sessions, "What was her role? Well, she didn't go up to her bedroom and watch soap operas while we sat downstairs and charted the course of the campaign."

She was also credited with determining the timing of television advertising during the campaign--against the advice of staff--and with having strongly encouraged Hughes to pick J. Joseph Curran Jr. as his running mate. Asked about Curran, she said she likes the lieutenant governor but the choice of running mate was her husband's.

She sits in on most press conferences and is said to follow up with analyses of questions and the governor's responses. She has a staff member who attends all meetings of the governor's senior staff.

She took control of scheduling all outside activities through the "box," and after all the aides have had their say, she is the last person Hughes hears from about whether to attend a Baltimore businessman's reception or a Prince George's County Democratic roast. During the reelection campaign, that was a particularly vital role.

Pat Hughes said, "We make decisions together and it's not a question of anyone dictating to anyone else." But those who know both Hugheses say the governor is willing to agree to attend anything if the staff beseeches him--it is Pat who curtly draws the line.

Last year, when Ejner Johnson, the current chief of staff, tried to set up a reception for county officials during the Maryland Association of Counties convention, it was Pat Hughes who ordered it canceled at the last minute because the reception had not be processed through "the box." Said Johnson: "She told me I overstepped my bounds. She was right, it hadn't gone through the proper procedures."

When Harry Hughes became governor in 1979 following the scandal-dominated years of former governor Marvin Mandel, he self-consciously set about showing that the government and politics would be conducted differently. There would be no more cozy political clique with a free run of the governor's office.

Pat Hughes felt the need to prove the same point at the mansion, the 54-room Georgian edifice that physically and socially holds a prominent position in the state capital.

"We were elected by people who voted to throw out what came before," she said.

The major public emphasis was immediately placed on the so-called reinstallation of rooms at Government House--fashioning each into authentic colors and furnishings of various eras, with the help of the Maryland Historical Society.

Pat Hughes also helped create a Friends of Government House and the Government House Trust to ensure that the efforts would continue under future governors. She received widespread praise for this.

But proving the point that times had changed also involved the way the house was run. Days after Hughes' 1979 inauguration, Pat Hughes let it be known that casual drop-ins at the mansion by staffers and legislators would not be tolerated. It was now, she said, The People's House, for all Marylanders to use, not just for the political types who frequent Annapolis. To document her point, she keeps a detailed list of the 179 official receptions involving groups all over the state held in Government House since the Hugheses moved in.

Pat Hughes said the new policy, agreed to by her husband, was designed to restore dignity to the mansion and make it more inclusive. She instituted official guest lists and formal invitations, receiving lines, and some mandatory black-tie receptions.

Said William Boucher III, a Hughes friend and political fund-raiser during the last five years, "She wanted to do it right."

But strict adherence to doing things right produced detractors--many remain after five years--and complaints that it is too formal, too rigid, so consciously above politics as to make the political world, and politicians, in which it is located, seem squalid.

Reliance on approved guest lists caused unexpected political fallout, particularly when one legislator stomped off from a reception in a huff after his date, not on the approved list, was turned away at the door.

Hughes later soothed the legislator, and rigid adherence to the rule seemed to relax somewhat after that incident and during Hughes' 1982 reelection campaign.

In some instances, people had to wait outside in the rain or snow because they arrived a few minutes before the appointed time.

"She tries so hard to do this right that sometimes it comes out wrong," said one person who has seen early-arriving visitors gather in front of the wrought-iron gate.

Several legislators boycotted the annual reception for the General Assembly this year because of a black-tie requirement. One of them, Republican Sen. John Cade, the minority leader, said, "I found it would cost me $60 to rent a tux and that seemed too much for one night. I resented the fact that they were made black-tie affairs." Cade is otherwise complimentary about her efforts to restore the mansion.

Said one Hughes adviser: "I think you could probably say that she has hampered his dealings with legislators. People come to these things to rub shoulders with the governor and the set-up has made it much harder to do."

Both the governor and his wife said the formal occasions were enjoyable for them and most of their guests, but they became an issue during the 1982 campaign. Hughes' GOP opponent, Robert A. Pascal, frequently used the formality of the mansion to criticize Hughes as aloof.

Pat Hughes responds: "A lot of people are pleased with the turnaround in image, in substance. Nobody fusses about a receiving line at the White House. I think flak comes with the territory."