Montgomery County Executive James P. Gleason, whose independent ways and frequent criticism of the size and complexity of government marked him as the skeptic among area politicians, announced yesterday that he will retire from public office when his term expires at the end of the year.

Gleason said he would seek a career writing about government, among other things, and about how the bureaucracy "chokes energy and creativity and stultifies your hopes."

Gleason removed himself as a potential Republican candidate for governor of Maryland, leaving that party with few serious contenders. His departure from the county executive's race is expected to set off what one county politician called a "mad scramble" for that powerful office.

Gleason is the county's first executive under the council-executive system begun in 1970 and his election to a third term this year, had he decided to seek re-election, was considered a strong likelihood.

Gleason became the third Montgomery County officeholder to withdraw from political life over the past few months. All of them - Gleason, and council members Dickran Hovsepian and Norman Christeller - cited the increasing demands of government in time and energy as leading factors in arriving at their decisions.

Gleason, 56, appeared at his press conference yesterday characteristically disheveled, with his shirt collar open and his tie loosened. He said he wanted "to be able to express the things inside fo me through writing that I can't express through government."

Gleason said he was not fleeing public office in frustration but that he was bothered by the "tremendous cmpolexity involved in the process" and "the ungodly hours" demanded of his job.

"I am not frustrated," Gleason said in an interview after his announcement. "I have done all I can do personally and now I want to do something else. Maybe I can get my thoughts into government through my writing. Whether its a play or a novel or an article is irrelevant," he said.

Gleason presided over one of the nation's wealthiest counties, where an overwhelmingly white, well-educated population of half a million pushed the county's resources to their limit. A moratorium was imposed on sewer hookups in 1970 because of these demands. And while the moratorium put the brakes on growth, it forced the price of housing to astronomical levels. The average cost of a house in Montgomery is now more than $70,000.

In his two terms in office and his short term as a County Council member, Gleason often found himself immersed in long and complex disputes with the federal government, the other area jurisdictions, the Democrat-controlled County Council and his own constituency. These confrontations served to jell Gleason's reputation for being a maverick, an independent thinker and a rather petulant man.

Last year, in order to force the federal government to guarantee that the Wheaton-Glenmont subway line would be built. Gleason withheld $32 million of Montgomery County's share for the construction and operating costs of the rail transit system. Gleason released those monies last December without receiving a guarantee, but with prospects looking better for the completion of that line.

In the early 1970s when it became obvious that the Washington area needed an additional sewage treatment plant and leaders of the area jurisdictions were arguing over where the new plant would be located. Gleason offered to place it in Montgomery County, a move that angered county residents.

Later Gleason sued that District of Columbia - the second time he brought the county into a legal suit over the sewage issue - in order to ensure the District's support for a plant to be built at Dickerson.

He was one of staunchest early supportes of a regional Metro system and was an early chairman of the Board. He battled fiercely in the late 1960s for two Montgomery County lines - and won. He just as fiercely opposed a single-line proposal that would have had the Metro line in Montgomery going only along the B&O right of way beyond Silver Spring. It was because he won that battle in the '60s that he fought so hard to save the Wheaton-Glenmont line last year.

"Jim's not easy to work with," said one longetime associate in Metro affairs, "but it's impossible not to respect him. And he is a gambler. He loves to push things down to one throw of the dice. Then if it doesn't look good, he just lays the dice down without ever rolling them. . . . He's won a lot that way but it's tough on everybody else."

Gleason often was the odd man out in various intragovernmental affairs. Last year he abruptly pulled out a coalition of county executives formed to lobby the Maryland legislature, saying he thought it was a waste of time.

His open contempt for the everrising expenses of an ever-dwindling county school system sometimes made him public enemy number one of school officials and teachers.

He was exhilarated by confrontations with his opponents. Once, when he attended a Fraternal Order of Police meeting in Gaithersburg, he told the police in no uncertain terms what he thought of their latest pay proposal. As the evening wore on and the questions got nastier and nastier, Gleason appeared to smile more and more as he repeated his view that the county could not afford larger paychecks.

In his first term, Gleason worked at reorganizing the county government. He brought most of the social service agencies under the umbrella of the Department of Himan Resources. He also created an Office of Consumer Affairs, Women's Commission and Office of Drug Abuse.

Most of his beliefs about streamlining goverment, Gleason said, date back to his Jesuit education at Georgetown University. That education, he said, which stressed philosophy, logic and ethics, "equips you to look at the essence of things and not to be distracted by the incidentals."

In his office Gleason has copies of James Joyce, novels and various books on his ancestral country, Ireland, in addition to a copy of Modern English Usage and the 1976 edition of Writer's Market. On a coffee table, a English textbook he has had since his Georgetown days lay open to an essay on literature by John Henry Newman amid copies of various county reports.

Gleason said he will seek a teaching position to supplement his income as a writer.