When asked recently why a colleague had voted against a defense program, Sen. John Tower snapped: "He abuses the right to be stupid." When Tower points the Green Bullet (his 1972 Dodge Charger) toward Texas next year, the Senate will lose one of its tartest tongues and sharpest minds.

Tower, whose office is a few yards from what was Sen. Henry Jackson's office, will not quite say so, but he might have reversed his decision to retire if he had not announced it before Jackson died.

In the Senate, as in many other institutions, 20 percent of the members do 80 percent of the work. For two decades Tower and Jackson have been two of the consequential 20. Since 1981 they have been, respectively, chairman and ranking Democrat on the Armed Services Committee. So in 1985 the Senate is certain to be without at least 15 percent (counting Howard Baker) of the 20, and even more of the heart of its leadership on military matters.

Tower has been in the Senate 22 years, 20 of them in the minority. In 1965-66 he had just 32 Republican colleagues. No senator, having been a chairman, can stand the thought of returning to the minority.

But there are 19 Republican and only 14 Democratic seats up in 1984, and today at least six Republican seats (Texas, Tennessee, Illinois, Iowa, North Carolina, New Hampshire) look more vulnerable than any Democratic seat. (Savor this, gender-gap scholars: some Republicans hope three women who are potential candidates--in Michigan, Rhode Island and Nebraska--can save Republican control of the Senate by defeating Levin, Pell and Exon.)

Tower thinks, probably rightly, he would have been reelected. Polls show him 20 percent ahead of his nearest rival; his approval rating is higher than ever. And he actually likes campaigning across his unreasonably vast and complex state.

However, the Texas electorate is 11 percent black, 18 percent Mexican- American, and as the 1982 defeat of Gov. William Clements showed, Texas Democrats have been reading the Republican book on organizing turnouts. Tower would not have needed to spend a nickel to build name recognition, so his campaign might have cost "only" $7 million. Texas has 19 media markets. Four cover 70 percent of the electorate, but the other 15 can turn an election like the one Tower won in 1978 by 12,227 out of 2.3 million votes cast.

Beyond a desire for a fresh and less draining life, Tower's decision to leave the Senate reflects dismay about the institution. Not long ago, he says, the Senate was more efficient and civil, in part because power was concentrated in a few persons, who received considerable deference--persons like Richard Russell, Everett Dirksen, Styles Bridges, Lyndon Johnson, Bob Kerr. There never was a record vote unless the leadership wanted one. That saved time and, even more important, prevented what has become common--the engineering of record votes, often on amendments that are going nowhere, often for grandstanding purposes, or to get responsible incumbents to make themselves vulnerable to irresponsible challengers by casting politically dangerous votes.

Tower thinks the Senate has "lost its corporate memory." Forty-three senators have been there less than six years, and 63 less than 10. Too many of the new members are "media creatures." They are frightened of politically awkward publicity. They increasingly share, or at least are inhibited by, the values of a press corps that sees itself as an adversary of established institutions.

Many younger senators assess their success by the media attention they receive. The easiest way to get attention is by challenging the executive on defense and foreign policies. That is one reason why, having awakened to the fact that it has delegated too much discretion to the executive, Congress began asserting itself in the wrong area--in defense and foreign policies, where it is least equipped to cope, and where deference to the executive is proper.

Because President Reagan's defense buildup has been the most important public business since 1981, since 1981 Tower and Jackson have been, with Baker, the most important senators. As Reagan contemplates the likely mixture of pain and pleasure in a second term, he must be imagining how hard it will be for him to deal with a Senate with those three men missing.

When Tower leaves, the Senate will lose half its contingent from the London School of Economics. (Patrick Moynihan is the other alumnus.) Tower, a former professor, wants to do some teaching. That is good news for American education. Some students are going to learn something, and their teacher will not grade on a curve.