Common Cause of Virginia accused Sen. John W. Warner (R-Va.) yesterday of reneging on a campaign promise by voting to increase the amount of outside income a senator is permitted to earn.

Warner responded by saying he changed his position after hearing some of his poorer colleagues talk about the hardships of maintaining two homes and sending children to college on their $59,500 Senate salaries. "I was persuaded to give them the latitude," Warner said.

Judy Goldberg, executive director of Common Cause for Virginia, at a news conference in Richmond, criticized both Warner and Sen. Harry F. Byrd Jr. (I-Va.) for giving themselves "a backdoor pay raise" by voting to increase the limit on outside income from $8,625 to $25,000 a year.

Both seantors are multimillionaires.

(Maryland's senators, neither of whom is a millionaire, divided on the question. Republican Charles McC. Mathias voted to raise the limit, and Democrat Paul S. Sarbanes voted against it.)

Goldberg said that Warner signed a Common Cause questionnaire last Sept. 1 saying he would "oppose any efforts to weaken or modify the 15 percent limit on earned outside income."

Referring to that pledge, Goldberg asked, "What trust can Virginians place in a junior senator who at the first opportunity reneges on a campaign promise?"

Bill Kling, Warner's press secretary, said that despite Warner's vote on Wednesday, he will continues to follow a policy "that he will not accept honorariums."

Warner and Byrd voted with the majority Wednesday when the Senate voted 54-to-44 to reaffirm an earlier voice vote to raise rhe income limitation. The $8,625 limit was set in 1977, in return for a salary increase of $12,9009

Neither Warner nor Byrd was present for the voice vote, which was widely critized as an indirect way for senators to expand their income.

Goldberg said a questionnaire sent to Byrd when he last ran for office did not contain a question about outside income. But she said that in Virginia, where fiscal conservatism is popular, "I thought we could certainly count on their votes."

An aide to Byrd said yesterday that he voted against imposing the limit in the first place, in 1977. "He doesn't get paid for his sppeches," the aide explained, "so it doesn't make any difference to him."

Warner's spokesman Kling noted that Warner's change of mind was influenced by the signing last October by President Carter of the Ethics and Government Act of 1978, law that has prompted a number of top officials to threaten to leave the government contending its provisions are too strict.

Warner also found a Washington Post editorial in favor of raising the income limit "a compelling argument" in behalf of the proposal, Kling said.

Goldberg said later she was unmoved by Warner's arguments. "The responsibility of Virginia's senators ought to be to their constituents - not to their Senate collegues - no matte how tough it is to get by on salaries that are four or five times greater than thos earned by most Virginian," she said.