The man who emerged this week as the leading contender for the chairmanship of the powerful House Budget Committee is a black Baptist minister from Philadelphia who is as comfortable around the back alleys of politics as he is in the pulpit.

A shrewd political operator, they call him in the House, and all the more so this week. When the dust settled after Tuesday's fight in the House Democratic Caucus over whether to change the rules affecting the election of the Budget Committee chairman, Rep. William H. Gray III (D-Pa.) was standing tall.

The caucus defeated a move by two senior committee members, Reps. James R. Jones (D-Okla.), the current chairman, and Leon E. Panetta (D-Calif.), whose terms on the committee are expiring. They had wanted an extension to let them run for chairman next month. But other House Democrats said no, opening the way for a new chairman of the committee that will draft a Democratic alternative to President Reagan's forthcoming budget.

Gray, just reelected to his fourth term, has been campaigning quietly for the Budget chairmanship for months, and Tuesday's vote not to change the rules for Jones and Panetta was a sign that his effort is paying dividends. If elected to the post in January, he will be the most powerful black in Congress.

"His greatest strength is probably best characterized by the kind of campaign he's run," a House leadership aide said. "A candidate who no one took seriously, he's done all the right things. He's reached out to places no one thought he could get, and has turned himself into the leading contender."

Gray has demonstrated that kind of skill since he came to Congress. In 1978 he defeated Rep. Robert C. Nix (D-Pa.) in a tough primary fight, aided by Andrew Young, then ambassador to the United Nations. When Gray arrived here, he mounted a campaign that won him a seat on the Budget Committee as well as the Democratic Steering and Policy Committee. From that slot, he was able to do favors for other freshman Democrats seeking key committee assignments.

"He's a man who knows how to walk into a situation, see an opening and insist on it," another leadership aide said.

Gray left the Budget Committee in 1981 and rejoined it in 1983. Last year he played a key role as a liaison between the panel and other House liberals and the Congressional Black Caucus, helping to convince members that the more centrist budget resolution coming out of the committee was the best deal they could get.

Colleagues say Gray's strength is in understanding how to put a coalition together, how to take care of the interests of other House members, and the importance of finding a consensus within the institution.

"Bill's shown flexibility and a mastery of the budget process," said Rep. Charles W. Stenholm (D-Tex.), a leader of the conservative "Boll Weevils" in the House. "He's very articulate and he's always been fair."

Stenholm said Gray would be an "excellent" chairman, even though he is committed to fellow Texan Martin Frost. His praise of Gray is indicative of Gray's ability to reach beyond his liberal base to build support for his candidacy. Among those who have announced their support for him is outgoing chairman Jones.

Frost is the only other announced candidate for the Budget Committee chairmanship, but several House Democrats are considering running.

Some House members say that although Gray is a smart politician he is shaky on details of the budget and the economy. They say he did not actively participate in the nuts-and-bolts work on the committee and they voice concern that he might not be a match for the White House or Senate Budget Committee Chairman Pete V. Domenici (R-N.M.) in the long negotiations expected over the next budget.

"I think I would bring a good knowledge of fiscal policy," Gray said yesterday.

But he made clear that he thinks the chairman's job is not simply to know fiscal policy, but to sense the "emerging consensus" in the House and respond to it. As he reads it now, Gray said, the consensus is for tough action to reduce the deficit, further spending cuts "made equitably in all areas" and a resolve that any new taxes must come from Reagan.

Gray said he believes the fact that he is black will not materially affect the vote on the committee chairmanship, even though there were racial overtones to Tuesday's debate about changing the rules in the caucus.

Gray is aware that many people wonder whether he is too liberal to head the Budget panel at a time when further cuts in social programs are likely.

Asked about that recently, he replied: "It was President Richard M. Nixon who went to China."

That kind of political sense has put Bill Gray into the front ranks of the fight for one of the most powerful positions in the new Congress.