Former Virginia Republican congresssman M. Caldwell Butler of Roanoke today abandoned his campaign for the GOP nomination for governor next year, saying he was "not comfortable running against my friends."

"It looks like a very confrontational campaign," Butler said in an interview as he announced his withdrawal.

Butler said it is likely others may enter the race, which has been focused on former Fairfax legislator Wyatt B. Durrette, former state attorney general J. Marshall Coleman and Butler. Others being mentioned include Rep. Stan Parris of Fairfax, Rep. Herbert Bateman of Newport News and Josiah Bunting, president of Hampden-Sydney College in central Virginia.

For the past four months, Butler, who drew much attention for his vote on the House Judiciary Committee to impeach Richard Nixon 10 years ago, has sought to portray himself as an electable alternative to either Durrette or Coleman, the two apparent GOP frontrunners.

"It is now apparent that this is not the universal Republican view," Butler said in a letter sent to his supporters made public today.

" . . . My experience . . . and my track record of 11 successive victories over 20 years was a good indication" of his strength, wrote Butler. He retired from Congress in 1982 after serving 10 years. Both Coleman, the party's candidate for governor in 1981, and Durrette, the nominee for attorney general, lost to Democrats that year.

Butler's withdrawal is the second major change this year in the rapidly developing campaign to succeed Democratic Gov. Charles S. Robb, who under Virginia law cannot succeed himself.

In May former Republican governor John N. Dalton of Richmond withdrew from consideration. Dalton, recovering from cancer surgery, virtually was assured of the nomination had he wanted it.

The former governor, who praised Butler's entry into the race, could come under pressure to change his mind about the 1985 race, some party leaders said privately. But a source close to Dalton said today that he is not likely to heed the call.

Durrette, now a Richmond lawyer, praised Butler and said his withdrawal "obviously is of major significance. He had attracted some considerable interest and encouragement. The decision that those who were supporting him now make . . . is of major importance for the next few days and weeks."

Coleman, now living in McLean, said, "I always had a gut feeling he wasn't going to go the full race. He'd like to be governor, but he comes from a recent era when it was always agreed who the Republican candidate would be. He would have been a very good governor. In weighing what you have to do to get there, he didn't think it would be worth it."

Butler, who has been mentioned several times for statewide office but never mounted a formal campaign, had proposed frequently that Durrette drop back and run for lieutenant governor on a ticket with him. Durrette ruled out such a move three weeks ago.

"I am greatly concerned that the campaign . . . will deteriorate and jeopardize our chances no matter who emerges with the banner; and I would prefer not to be a part of that," Butler's letter said.

Butler also said he "was prepared to wage the total campaign necessary, from nomination to general election," but said "I simply do not have the time to spend the additional 10 months prior to the convention" to win the nomination.