James E. Whitehead Jr., one of two black Republicans who planned to vie for the GOP nomination to challenge Democratic Rep. Steny Hoyer in the fall, said yesterday he has decided instead to run for the state Senate from Prince George's County.

Whitehead, 24, an aide to Rep. Jack Kemp (R-N.Y.), said he was switching races because he does not think he could beat Hoyer. But Whitehead also said he had been pressured by national party leaders to drop out in favor of the Rev. Perry Smith.

Whitehead said he now plans to challenge incumbent Democrat state Sen. B.W. Mike Donovan, whose seat also is being eyed by Democrat County Council Chairman Gerard McDonough.

Smith was one of a dozen black ministers from the county who converted to the GOP this year. They were promptly feted at the White House by President Reagan and other GOP leaders. In 1971, as a Democrat, Smith ran unsuccessfully for the County Council on a slate headed by Hoyer, who ran unsuccessfully for county executive.

Smith has hinted that officials at the Republican National Committee have told him he might get financial support in the primary if he runs unopposed. Whitehead's decision apparently clears the way for that funding, although a spokesman at the RNC would not comment on that yesterday.

"I don't believe I have enough time to raise the money to beat Hoyer," said Whitehead, who said much of his $12,000 campaign fund was raised at an April 3 party that featured singer Issac Hayes.

Dr. Allan Levey, Maryland GOP chairman, hailed Whitehead's decision as "an opportunity to win a lot of races in Prince George's County. It would be a shame to have two good candidates running for the same office."

Smith said he had not been told of Whitehead's decision, which is to be formally announced today. Smith said he plans to announce his candidacy on Tuesday. He said RNC chairman Richard Richards might attend, adding "It is my understanding that they will make a presentation of some funds."