Five-term Rep. M. Caldwell Butler (R-Va.) announced yesterday he will retire from Congress next year, opening the way for J. Marshall Coleman, the recently defeated Republican candidate for governor, to seek the Roanoke area's House seat.

"As soon as it became known that he Coleman would be unemployed after January, I advised Marshall to include this among his options," Butler said. Coleman, a native of Butler's 6th Congressional District, declined to commit himself to the race, but said, at Butler's request, he "won't close the door to it."

Butler, who said he was homesick and frustrated with his life in Washington, said he is not planning to endorse a successor to the seat he has held since 1972. "A lot of potential candidates have expressed interest and I wanted to give them all plenty of time to jockey for position." Butler said. "The line is forming."

Coleman, who leaves office as state attorney general Jan. 16, said he is reluctant, for personal and professional reasons, to consider another political campaign so soon after his Nov. 3 defeat by Democrat Charles S. Robb.

Coleman lost the congressional district to Robb by about 2,000 votes. Even so Butler said yesterday Coleman's "base is pretty secure" in the traditionally Republican area, which runs from Harrisonburg to Roanoke. The 39-year-old Coleman, who was born in Staunton and grew up in neighboring Waynesboro, would face some obstacles in winning the GOP nomination, Butler said.

Coleman faces a substantial deficit from his gubernatorial campaign -- estimated at more than $500,000 by some accounts -- and that he has "some fence mending to do back here" in the district, the congressman said.

State Sen. Ray L. Garland of Roanoke said yesterday that he would "almost certainly be a candidate" and quickly sought to establish that he would be a stronger nominee than Coleman. "Most Republicans don't do well in the Roanoke area," said Garland, noting that that was his base.

Butler said other Republicans who have expressed an interest in succeeding him are Del. Arthur R. (Pete) Giesen Jr. of Verona and former delegate Ray Robrecht of Salem.

The GOP has held the sixth since 1952, but Butler said the party cannot asssume the seat is safe. "We now have a functioning two-party system in Virginia, and a weak candidate cannot be expected to elected automatically," he said. "It's a strong Republican district, every bit as secure as the governorship," Butler added, displaying the dry wit that has made him famous in the House.

Democrats C. Richard Cranwell of Vinton and A. Victor Thomas of Roanoke, both members of the state House of Delegates, said they would consider running. Also mentioned by Democratic leaders were State Sen. Dudley J. (Buzz) Emick Jr. of Fincastle, who was an unsuccessful candidate for his party's nomination for lieutenant governor this year, and John S. Edwards, whose term as U.S. attorney in Roanoke expires next week.'

Butler, 56, appearing with his wife, June, at a press conference at the Patrick Henry Hotel in Roanoke, said that he is in good health and plans to practice law there.

There was some speculation, however, that the move would put Butler in a position to run for the Senate next year, should incumbent Harry F. Byrd Jr., the chamber's only independent, decide not to run. Butler declined to comment yesterday on those suggestions.

Butler considered running for the Senate in 1978, following the death of GOP nominee Richard D. Obenshain. He then withdrew in favor of John W. Warner after Gov. John N. Dalton endorsed Warner.

On Capitol Hill, Butler is viewed as a witty, articulate conservative. He was among the Republicans on the Judiciary subcommittee who voted to impeach then-President Richard M. Nixon and more recently played a major role in leading opposition to extension of the 1965 Voting Rights Act.