The Maryland Appellate Judicial Nominating Commission has rejected House Speaker John Hanson Briscoe as a candidate for a vacancy on the Court of Special Appeals.

The decision to bypass Briscoe has so disappointed Acting Gov. Blair Lee that he said last night he will change today the rules under which Briscoe failed to qualify.

Lee said, however, that will not tamper with the "integrity of the process" that resulted in certification as qualified of only two lower court judges, both from Prince George's County.

Briscoe was one of four applicants for the vacancy on the state's second highest court, but the 12-member selection committee certified as qualified only Circuit Court Judge James F. Couch Jr., and District Court Judge James F. Rea. The fourth applicant was State Del. John J. Garrity (D-Prince George's).

Lee is believed to be leaning toward the appointment of Couch, 60, to fill a seat being vacated at the end of the month by the retirement of Judge Jerrold V. Powers.

The nominating commission was created by an executive order of Gov. Marvin Mandel in 1970. Present rules require that no fewer than two names be forwarded to the governor. It is from those that he selects one to fill a vacancy.

Lee said the screening committee's decision to submit only two names to him for the vacancy, coupled with a similar action last month by a lower court selection committee "leads to a breakdown of the system."

Lee said he will "rescind and reissue" the executive order creating the commission, so that future decisions will require that screening committees nominate no less than two candidates for trial courts in small counties, three names for trial court vacancies in large counties and Baltimore, and a minimum of four names for vacancies on the two appellate courts.

Lee also said last night that he has decided to go ahead and name District Court Judge Vincent J. Femia to the vacancy on the Circuit Court bench in Prince George's County.

While Lee praised Femia s "a good man," he said it nonetheless was a "charade" for Femia's name to be the only one offered to the governor for appointment.

The judicial screening committees were established with the stated intention of assuring that only qualified persons were nominated for judgeships in the state.

But previous lists of nominees recommended by such committees invariably included at least one person with good political connections. It was a process that enabled Gov. Marvin Mandel to reward friends with judgeships while also allowing him todefend the selections as nonpolitical. Lee continued to appoint Mandel friends during his tenure as acting governor because he said he felt he had to honor Mandel's wishes.

The most recent example of the procedure working in that manner was the selection by Lee this summer of Alan M. Wilner to a newly created seat on the Court of Special Appeals.

Wilner, who was Mandel's chief aide, was included among a list of qualified candidates, and Lee named him to the post.

The vacancy created by the long-expected retirement of Judge Powers was so widely believed to have been "reserved" for Briscoe that Briscoe, to avoid the potential appearance of a conflict of interest, did not vote on the legislation approved last spring that added the 13th seat to the court.

Both Briscoe and Lee agreed that there was no promise to Briscoe that he would get the vacancy, but Briscoe acknowledges that he told Lee of his interest early this summer.

Lee said last night that he was "totally surprised" by the failure of Briscoe to win certification. "I was frankly thinking pretty positively of appointing Briscoe," Lee said.