Mimi Lee, 61, dons her bathing suit four times a week and turns up poolside for a sport that has caught on in recent years like a house on fire: masters swimming.

Lee, a Silver Spring resident and wife of former acting governor Blair Lee, competes in more than 20 speed and endurance events. And she is ranked in the top 10 in her class nationally in nearly every one of them.

"My times? Oh, my times are not very good," she says. "People will die laughing when they see my times. It's like the old story of the dancing dog. The dog doesn't dance very well. The amazing thing is that he can do it at all."

Masters events are only for over-25 swimmers and Lee competes in the 60-64 age group.

Art Smith, president of D.C. Masters, the club for which Lee swims, says her ranking is not just a matter of outlasting the competition. "In masters, times are compared nationally," he said. "To be ranked in the top 10 in even one event is good."

It's getting more difficult for Lee to dismiss her swimming feats with modest protestations. Swim Swim magazine, one of two national magazines catering to masters swimmers, recently announced that Lee and fellow D.C. Masters member Kelly Lemmon (men aged 65-69) made its 1981 all-star team. Team members are, the magazine says, "the most accomplished all-around masters swimmers in the country."

Like all veterans of the sport, Lee says she has witnessed a tremendous growth in participation.

"I can't quote you numbers," she says. But "it's definitely on the upswing. . . . The population is getting older. This is for people over 25 and there's a lot more of them. It masters swimming has no place to go but up."

Spawned by a physical therapy program started by California physician Dr. Ransom Arthur, masters swimming has grown from the handful of swimmers who attended the first national meet in 1971 to the 20,000 currently registered with the national organization--United States Masters Swimming Inc. According to Smith, membership in the national group is growing by about 8,000 a year.

Dave and Eileen McAfee founded the D.C. Masters in 1972 and have been officers of the club for most of the years since. In the early '70s, Eileen McAfee recalls, 250 was thought to be a large attendance for a national meet. At the 1981 Long Course Nationals in Canton, Ohio, nearly 1,000 swimmers participated.

Similarly, Dave McAfee, an avid competitor in the 70-74 age group, could find only six meets a year when the sport was organized. He now attends about 12 a year.

Locally, about 300 swimmers constitute the six clubs registered with the Potomac Valley Masters Swimming Committee. The committee's jurisdiction includes Prince George's, Montgomery, Arlington and Fairfax counties, the cities of Alexandria and Falls Church, and the District of Columbia. Several hundred additional swimmers train with organized masters groups that are not members of the committee. Beginning at age 25, masters compete in five-year age groups with a sub-masters category for 20-24-year-olds.

Although Potomac Valley is one of the smallest geographic jurisdictions, it is the third largest in the country in terms of membership, an impressive statistic when such states as California and Florida--where swimming ranks high--are considered.

The D.C. Masters club is the heaviest contributor to the Potomac Valley's strength. With a membership roster of 150, it is the largest club locally and one of the most successful nationally. It consistently places in the top four in any meet it enters.

The McAfees are a big part of D.C. Masters' success, say many area swimmers. They point out that many of the all-volunteer clubs languish for lack of administrative manpower. Referring to the "500 hours a year" he puts in, McAfee says, "most clubs don't have a permanent treasurer to keep it going."

"I joined D.C. Masters because it's large and I get good service," says Lee. In addition to the seven meets the club runs each year, times are compiled, records kept, dues collected and medals awarded with great efficiency. "It has excellent management," she says.

But good management and hefty membership are just part of the story. The D.C. club boasts some extraordinary swimming talent. In addition to all-stars Lemmon and Lee, 30 other members are ranked in the top 10 nationally. All 14 of Potomac Valley's 1982 Outstanding Male and Female Swimmers are members of D.C. Masters.

When asked, masters say that health and fun are the primary reasons for their dedication to workouts, but competition is the linchpin of the sport.

"The impression you get when you start," says Lee, who first competed in the 50-55 age group, "is that you're trying to improve your times. Then you notice you're swimming against someone."

At big meets the competition is as fierce as it is in any other sport.

"Competing is important to me. I find I'm more likely motivated to get in there and beat my brains out when I have a goal," says President Smith, a Potomac Valley Outstanding Swimmer for 1982 in the 40-44 age group.

Lee notes that among masters swimmers, competition can be as hyped or as relaxed as the individual wants. Dave McAfee says that 40 percent of the D.C. Masters who work out regularly do not compete at all.

Even after 10 years of training, Lee finds she still is improving, a fact she attributes to good coaching. This year in the hour-long swim sponsored by D.C. Masters, she set a new personal best by swimming 135 laps.

Masters swimmers drive themselves hard and some arrange their lives around pool schedules. Stories of 5 a.m. swims that dictate 8:30 p.m. bedtimes are not unusual. Such behavior sometimes is viewed as eccentric by those on the outside looking in.

Lee says her nonswimming friends "think I'm insane. They wouldn't go out at 9:30 at night to swim."