The Rev. B. Sam Hart, the evangelist who withdrew his controversial nomination to the U.S. Civil Rights Commission last week, alleged yesterday that homosexuals and Communists have "infiltrated" the highest levels of the government and the media.

Hart, speaking to the Baptist Ministers Conference of Washington, D.C., and Vicinity, said he withdrew his nomination because the press had begun to fabricate derogatory stories about him. He said that even if they were proved wrong, the stories would create doubts about him in the minds of people he meant to help by serving on the Civil Rights Commission.

"We have allowed in our generation, let me say, you would be surprised how many, many Communists have infiltrated into the highest areas in government circles and into the highest areas in the press," Hart said in response to a question from one of the ministers.

"You would be surprised," he said, continuing the reference to government and the media, "how many homosexuals and homosexual sympathizers, those people, have been fed the line that homosexuality is a civil right."

In the view of those government and media officials he was criticizing, Hart said, "a man is permitted to be a homosexual like we are permitted to be black." He said he believed in civil rights for everyone, but that he believed homosexuality was immoral and disapproved of its public display. Hart delivered his remarks from the pulpit of Trinidad Baptist Church at 16th Street and Benning Road NE.

Hart, who was applauded throughout his speech, was asked by a reporter as he left the church if he has evidence of homosexuals or Communists holding important posts in the government or media. The Philadelphia minister said he would not give proof and did not feel the need to do so, after "the things the press has said about me."

Since Hart was chosen as the Reagan administration's nominee for the Civil Rights Commission last month, he has come under constant criticism from homosexual, civil rights and women's groups for opposing civil rights for homosexuals, busing to integrate schools and the Equal Rights Amendment.

The controversy mounted when it was reported that Hart had not been a registered voter since the early 1960s and registered as a Republican only last Nov. 4. In addition, on Nov. 7 a $100,000 Small Business Administration loan to Hart's broadcast company was declared in default. WYIS, a religious radio station owned by Hart, has also been found to owe $4,400 in taxes.

Hart, who said he has a worldwide evangelical radio show, said press reports of his radio station's tax debts neglected to say that those debts were for the current year and he has until March 15 to pay the bill. He said reports that he owed $23,541 to a minority business loan program failed to mention that a "new arrangement" has been made to handle the debt and his payments are now up to date.

Hart's speech--originally scheduled to be followed by a vote on whether the conference would endorse his nomination--began with a warning that America is now in a "critical and dangerous day" when the word of the church is no longer valued on serious issues facing the nation.

Hart cited his views on women's role in society as an area in which the press had distorted his beliefs. He said he believes in a traditional family in which a man is the head of the household, women take responsiblity as mothers and the children are "raised in subjugation to the parents."

In answer to a question from the ministers, Hart said he would not have withdrawn his nomination if only for pressure from the press about his stands. But he said that the stories being made up about him created doubts about him and made for division among blacks over his appointment when he was going to the commission to help blacks.

Hart finished by saying his advice to President Reagan's next nominee would be: "Fellow, keep your mouth shut until it's all over."