CHESTERTOWN'S finest hour was some 250 years ago. Then it was an important Maryland seaport and its leading citizens were busy shaping American freedom, enlisting in its navy and even holding their own "tea party" in the harbor. Rich merchants built Georgian and Federal homes topped with widows walks along its waterfront and ladies of fashion walked Water Street under their parasols.

But Baltimore harbor soon surpassed Chestertown's, and industry followed the ships across the bay. And what could be more fortunate for us 20th-century visitors who, beause of this, can take a look at a town suspended in the past?

At 23 High St. stands the White Swan Tavern, the reason why more people are getting to know Chestertown these days.

The Swan is a jewel of a tavern which has been lovingly restored by Horace Havemeyer Jr., a New Yorker who fell in love with this part of Maryland and opened the tavern as a bed and breakfast last April. No detail has been overlooked to make it perfect. Beginning with an archaeological dig to uncover clues to its earliest history, Havemeyer has spared no research or expense to turn this old building, whose history antedates 1733, into a well-appointed overnight stop.

The drawback is that only breakfast is served--but such is the charm of this tavern that you don't mind too much having to go out and scavenge for the other meals. There are only five bedrooms, but they are so large that you could put two or three motel rooms in each. The bedrooms all have huge tester beds and fireplaces. In each beautifully furnished room the original floorboards shine with a patina no electric waxer could give to modern wood. A bottle of wine and a basket of fruit greet you on the dresser, and there is a small refrigerator in each room. In short, you feel cherished from the moment you open the Swan's massive front door.

Not all the furniture is antique, but when it isn't it has been lovingly reproduced by the hands of local craftsmen or their counterparts in New York. Havemeyer even journeyed to Ireland in quest of marble to match the original fireplace in the game room, after his dig turned up a scrap of the stone. He had the scrapings of paint chemically analyzed to come up with the truly surprising bright blue in the tavern room. Hand-crocheted throws warm several of the beds. Fresh flowers are everywhere. There's not a telephone or a TV in sight (except for a single phone at the reservations desk), but if you just can't do without your nightly video fix there's a TV hidden away in a cupboard in the parlor.

Shards preserved from the dig are displayed in a case near the tavern bar. There's also a wonderful painting, copied from the original in the St. Louis Art Museum, depicting sea captains carousing in a tavern much like this one. The painter, a self-taught Massachusetts man, may not have studied art with the experts but he surely observed some first-class inebriation in his day. The scene he recorded gives you pause to think what must have gone on in taverns like this near the waterfront--it really wasn't surprising to hear that the floor of the tavern room of the Swan was originally covered with sawdust.

If you can't plan your life far enough ahead to book one of the Swan's rooms, which are much in demand on the weekend, you can see it all anyway on a tour from 4 to 6 p.m. on Fridays. Pay $2 and you'll get the history and a blow-by-blow description of the renovation, plus tea and homemade pastries. The rooms themselves are $60-$70 (no credit cards, please).

Chestertown's Water Street, studded with beautiful 18th-century homes, is only a pleasant stroll from the Swan. The Custom House at Front and High streets was the center of sugar trade with the British West Indies in colonial days and is worth a detour to see. Ask the Swan for directions to some of the other interesting old homes.

Sixteen miles south on Rte. 20 is Rock Hall, a fishing port where the catch of the upper bay is unloaded. There are several seafood restaurants where the fish served has just been caught, but unfortunately the cooking relies heavily on "all-American fry." We picked Hubbard's Pier and Seafood on the advice of a passer-by and because we could see by the boats at the dock and the trucks awaiting the catch that the fish would be only minutes from the water. The crab soup was hearty and good, but the soft-shell crab, on the menu remarkably early, had been long in the deep fry.

Rock Hall claims that it was here that Col. Tilghman, an aide to Gen. George Washington, changed horses on his ride to Philadelphia with the news of Cornwallis' surrender, but today visitors are usually bent on seeing the wildlife either at nearby Remington Farms or Eastern Neck.

Remington Farms is a wildlife management research and demonstration area laid out on 3,000 acres of what was once the estate of Glenn Martin, the airplane magnate. It's open to the public except in hunting season, when shooting parties invited by the owners, Remington Arms, stash themselves away in the duck blinds. The idea the rest of the year is to show how it is possible to upgrade the natural habitat so that farming and hunting can both prosper without decimating wildlife.

Until October this will be a Peaceable Kingdom, with ponds and marshes chockablock with mallards and some wood ducks, all so close you could almost reach out of the car and touch them. Duck populations are declining in North America and that worries everybody from conservationists to sportsmen. Wildlife specialists are studying the problem in this preserve. You tour the place in your car, guided by a leaflet you pick up before the start. The ducks, as unafraid as pedestrians on Connecticut Avenue, waddle across the road from pond to pond with barely a look at approaching cars.

Eastern Neck National Wildlife Refuge, beyond Rock Hall on a finger of land jutting south into the bay, is a true refuge, a major feeding and resting place for migratory waterfowl. The first white man arrived here in 1650, and the wildlife people who administer this island of brush, foxtail, mud flats and freshwater ponds say it looks much as it must have then. Bring your binoculars, for wildlife here is harder to spot and far more wary.

The walking trail leads you through a woodland community in which dwell some of the Delmarva Peninsula Fox Squirrel, officially classified as endangered. Look up and you'll see their leafy nests in the trees above the trail. If you're lucky you might also spot the bald eagle which has been haunting this area recently, and you'll certainly get a look at the osprey busy building their nests at Durden Creek.

Walk across the marsh by the boardwalk at Tubby Cove and, from the lookout platform here, you can pick out black ducks in the water. Though the black duck is declining in the Atlantic flyway, the refuge is attracting them by creating new freshwater ponds. There's usually a pair within range of your glasses.

Visit East Neck, as the locals refer to it, after May 1 and you can bring a canoe and go crabbing on the mud flats of the recreation area. Crabbers who are without canoes can bring instead an inner tube with a bushel basket set within, walk out 200 yards in the shallow water and net a catch. July is high season, but the blue crabs begin to show as early as May.

All this exploring whips up the appetite, so we decided to try our luck in Chestertown at the Old Wharf Inn. It turned out to be a nice place to eat and watch the gulls swooping over the river, settling like sentinels on the pilings of the dock. This time we ordered Oysters Rockefeller which, though the spinach was not pureed and the bacon was missing, were plump and juicy and thoroughly satisfactory. The bread here is sinfully delicious, making self-denial a real test. There's a pleasant atmosphere and a splendid view.

Only one of Chestertown's nice old houses is open to the public now, but you can get into the Moore-Geddes-Piper house beginning in June from 1 to 4 p.m. on Sundays. It's worth the time since it is filled with handsome 18th-century furnishings and with an especially fine collection of Chinese Export teapots. One of the owners was William Geddes, from whose brigantine the Chestertown citizens tossed tea overboard into the river in sympathy with their Boston counterparts.

One caveat: The Ken Narrows bridge is still a mess with construction; on a Sunday afternoon in March we were boxed in for more than 10 minutes. When I rubbed the travel fetish I keep in my glove compartment, like magic an additional traffic lane opened in front of us and we were able to get through thereafter with no trouble. Without a charm and a bit of luck you can count on, however, you should figure on weekend delays at this bottleneck.