Richard N. Viets, a 25-year veteran of the State Department, has told colleagues he is retiring from the foreign service following the refusal of the Senate to confirm his nomination to become the next U.S. ambassador to Portugal.

President Reagan nominated Viets last January, but the diplomat ran into tough opposition in the Foreign Relations Committee from the ranking minority member, Sen. Jesse Helms (R-N.C.), who accused Viets of financial irregularities while he was serving as ambassador to Jordan from 1981 to 1984.

After five meetings involving an exhaustive interrogation of his financial records by the Foreign Relations Committee, Viets was cleared of all wrongdoing and his nomination approved in July 28, 12 to 5.

Among the five who voted against him, however, were two Democrats -- Sen. Alan Cranston (Calif.) and Sen. Christopher J. Dodd (Conn.). They made an unusual combination to vote on the same side as Helms and two other conservative Republicans.

Nonetheless, a committee report said the majority had found Viets "to be exceptionally well qualified to be our next ambassador to Portugal," and his nomination was sent to the Senate floor. There it sat until the Senate adjourned in the early Tuesday morning. No action was taken to bring it to a vote. This meant Viets would have to start all over again with hearings and a new vote in the Foreign Relations Committee when the Senate reconvenes in January.

Insiders say the irony of Viets' long ordeal is that he was finally done in not by his main conservative GOP inquisitor, Helms, but by Senate Majority Leader Robert C. Byrd (D-W.Va.), who refused to bring his nomination up for a final vote.

Byrd was reportedly as upset as Helms about one particular issue in Viets' record that had nothing to do with his performance as ambassaador to Jordan or Tanzania or his role as deputy chief of the U.S. missions in Tel Aviv and Bucharest.

Viets had not paid state income taxes after he entered the foreign service, and had changed his official residence first from the District of Columbia to Vermont until 1984 and then to Florida. Helms claimed the changes were made so Viets could avoid paying taxes here. Helms charged that Viets never owned property or actually lived in either Vermont or Florida, but that he did own a home in the District and in Virginia, "both of which levy income taxes."

Viets sought a tax ruling from Vermont authorites about whether he owed taxes, and was told that the authorities did not regard him as ever having had a legal residence in there, according to an Oct. 20 letter Helms wrote to Senate Foreign Relations Committee Chairman Claiborne Pell (D-R.I.).