Sen. Lawton Chiles (D-Fla.), a 17-year veteran of the Senate and chairman of its Budget Committee, announced yesterday that he will not seek reelection next year.

Chiles' surprise decision -- he had been actively campaigning for a fourth term, was heavily favored and had raised about $1.3 million -- is expected to turn the 1988 Senate election in the nation's fifth most populous state into a wide-open contest in both parties. It also gives the Republican Party a chance to pick up another seat in its battle to regain control of the Senate, which it lost last year.

Chiles, a moderate Democrat who spent 12 years in the Florida legislature before his election to the Senate in 1970, said at a Capitol Hill news conference yesterday that he discovered during a recent campaign swing that he no longer had "the enthusiasm" for another six-year term.

"There is a time for all things," Chiles said.

"What my inner voice told me was . . . to move over and let someone who had the enthusiasm and zeal take over."

Aides said Chiles was "burned out" by the rigors of serving in the Senate, especially the frustrations accompanying this year's drawn-out budget deliberations. Budget battles have occupied the Florida Democrat almost continuously since January, when he became the committee chairman.

A soft-spoken man who relied more on persuasion than confrontation in leading the panel, Chiles seemed particularly drained by the month-long negotiations with the Reagan administration that last month produced an agreement to reduce the fiscal 1988 budget deficit by $30 billion.

Chiles is the sixth senator -- and third committee chairman -- to announce his retirement before the 1988 election.

The others are John C. Stennis (D-Miss.), chairman of the Appropriations Committee; William Proxmire (D-Wis.), chairman of the Banking, Housing and Urban Affairs Committee, Daniel J. Evans (R-Wash.), Robert T. Stafford (R-Vt.) and Paul S. Trible Jr. (R-Va.).

Sen. Pete V. Domenici (R-N.M.), ranking Republican on the budget panel, said that a "common theme perhaps" of the retirements was "frustration with the process, the workload, and the frustration with our inability to get to problems in a timely way."

Chiles, 57, said his health was not a factor in the decision, reached over the weekend and announced to his staff yesterday morning. Although he underwent a quadruple bypass operation two years ago, Chiles said yesterday he was physically in "good shape" to undertake a campaign and another term, but mentally was not "looking forward to another six years."

Chiles, whose earned the nickname "Walkin' Lawton" for traversing Florida on foot in earlier campaigns, said he began to have doubts over the Thanksgiving recess as he completed the first 56 miles of another campaign trek.

"There was just a different feeling than there was 17 years ago," he said.

"What I found out was I wasn't looking forward to another six years in the Senate. I was challenged by the campaign, but I didn't have the enthusiasm" for another term.

Chiles' announcement is expected to set off a scramble in both Democratic and Republican circles, and makes a Democratic victory in Florida next November less certain. A recent poll by the Florida Democratic Party had Chiles leading his only announced Republican opponent, Rep. Connie Mack III, by better than 2 to 1.

State party chairman Charles Whitehead said he is confident the seat will remain in Democratic hands.

"I don't feel as confident now as I would have with Lawton, but I'm not that apprehensive about it," Whitehead said in a telephone interview.

Chiles' retirement appears likely to draw at least one other Republican, Rep. Bill McCollum Jr., into the primary contest with Mack.

A spokesman for McCollum, who served on the House Iran-contra committee, said the 43-year-old lawmaker is now "considering" a race that he had previously declined to enter "out of respect" for Chiles.

On the Democratic side, several candidates are likely to emerge, including Rep. Buddy MacKay, Rep. Dan Mica, Florida House Speaker Jon Mills and state Insurance Commissioner Bill Gunter.

Gunter was defeated for the Senate in 1980 by Republican Paula Hawkins, who lost her reelection bid last year.

MacKay's top congressional aide said that the Ocala Democrat is "going to look at it seriously," but that no decision has been made.

An aide to Mica said the five-term South Florida lawmaker, who has $400,000 on hand, is forming an exploratory committee and will make a decision by the end of January.