A reference by the speaker of the Virginia House of Delegates to black legislators as "boys" caused black leaders to erupt in anger today, widening a racial cleavage that is already threatening the Democratic Party's chances for a U.S. Senate seat this fall.

Del. Robert E. Scott, one of five blacks in the legislature, said he was insulted by the comment and called on Gov. Charles S. Robb and other party leaders to repudiate Speaker A.L. Philpott or risk black defections in the November elections to fill the seat of retiring U.S. Sen. Harry F. Byrd Jr.

"The governor assumes the speaker meant no offense," a Robb spokesman said later. "But he Robb is very sensitive to what was said and can understand the reaction to it."

Sen. L. Douglas Wilder (D-Richmond), who had recently denounced Philpott for what he said was showing blacks "the back of his hand," said the speaker's remark pushed him closer to launching an independent candidacy for the U.S. Senate, perhaps by next week.

The comment, published in today's editions of The Richmond Times-Dispatch, was made Wednesday at the annual shad planking, a traditional political gathering in the woods of Sussex County, after a 30-minute meeting in Richmond at which the four black members of the 100-member House of Delegates voiced displeasure over committee assignments and other grievances with Philpott.

"I've never had any problems with those boys," Philpott was quoted as saying. "They understand the system."

"Can you believe Philpott would say that? Can you believe it?" asked Wilder, the only black in the state Senate. "That just shows that what I call the magnolia mentality is still there . . . If he characterizes our black elected representatives as 'boys' in that instance, then he views them as boys in every instance. He should apologize to all Virginians."

Philpott, 62, a tough-minded 30-year House veteran who came up through the ranks of the Byrd organization, was on a golfing vacation in North Carolina today and could not be reached for comment. Longtime friends defended him, saying the use of the word "boys" was standard in Philpott's Southside where the phrase "good ole boys" is a badge of honor.

"He'd be the last individual in the world to say anything derogatory about them--about the delegates in the House," said Del. C. Hardaway Marks (D-Hopewell). "He refers to me as a boy," said the 61-year-old legislator. "He means nothing in the world that would be racial at all."

Wilder's outrage was joined by Curtis Harris, state director of the Southern Christian Leadership Conference, who fired off a telegram to the speaker, calling on him to "publicly apologize to each member or give your tenure."

Del. James Christian, a 63-year-old black delegate from Richmond, said he was "very shocked" by the remark and demanded that Philpott offer an explanation.

Del. Scott (D-Newport News) said if Robb and other party leaders don't "publicly and viciously" criticize the speaker, "it shouldn't be any secret to anybody if the Democratic nominee has problems in November.

"There is a well-established connotation about the use of that word," said Scott. That it was made at the shad planking, until recently an all-white, male event begun by the segregationist Byrd organization, made it even more offensive, he said.

The furor over Philpott's remark comes in the midst of a squabble among Virginia Democrats that has increasingly focused on Philpott. During the recently concluded 1982 General Assembly, a series of bills sponsored by black legislators, ranging from a Martin Luther King holiday to a ban on state tax exemptions for segregated schools, were killed in House committees. Wilder and other blacks blamed Philpott for stacking the committees with conservatives hostile to black issues. The blacks legislators have threatened an independent Senate candidacy against expected Democratic Party nominee Del. Owen Pickett (D-Virginia Beach).

The meeting Wednesday was requested by the four black House members to discuss some of their complaints about committee assignments and other problems directly with Philpott. The blacks today described the discussions as "very cordial" and "helpful," adding that the normally taciturn speaker showed no animosity toward them even if he gave them no commitments.

"We didn't feel we had changed a whole lot, but that the lines of communications were open," said Del. Benjamin J. Lambert (D-Richmond). "He's always been a gentleman in dealing with us," he said. "But then again, Virginia gentlemen have different views when they get out of sight."