WHEN HARRY Kulkowitz set off from the Bronx for his first job as a waiter in the Catskills, his father the tailorgave him some advice.

"He told me that people work 50 weeks a year for two weeks vacation," Harry explains. "For two weeks, let them be kings, otherwise you rob them of the whole year. That's still my philosophy."

Now here he is, still a good son at 58, following his father's admonition by running an inn and restaurant on Cape May called the Carroll Villa Hotel and the Mad Batter. Three springs ago, a friend and I blundered into the Villa without a reservation and lucked into a cancellation. We keep going back to soak up more of Harry's philosophy and Batter's food.

One Sunday, we set out for a marathon escape to Cape May, a small town of Victorian inns hugged by a clean, wide shoreline at the southern tip of New Jersey. We left Washington at 7:30 on a Sunday morning and made the 10:40 ferry at Lewes, Del., with plans to catch the last ferry back at 8:30 that night. I figured it would be no problem. Aboard the ferry, my thoughts turned to the swooping seagulls and bell buoys bobbing like mirages in the mid-morning mist.

The drive to Cape May village is about 10 minutes from the ferry landing, over the bridge where the fishing boats dock and past a few introductory Victorian mansions. We talked of buttermilk pancakes, crepes and eggs mornay.

Harry Kulkowitz, bespectacled and smiling over his graying Vandyke beard, greeted us on the porch, next to the blackboard heralding the day's fare, a moderately priced eclectic array of French, Italian, Jewish and Oriental dishes. I got a handshake; my friend Marsha got a hug.

Every inn has a personality, a presence, and Harry Kulkowitz permeates the Batter and Villa. Harry is a thin 5-foot-6. He walks slowly, with his body slightly bent forward at the waist, and he seems to be everywhere, hugging friends and seating guests. His face opens up often in a broad smile, and he seems calm. He isn't. He's full of nervous energy that he takes out on his inn and staff.

Eight years ago Harry was running a successful art gallery in Philadelphia when he quit the rat race to "retire" at the age of 52.

"I felt Cape May was ideal," he explains. "I saw it as the new Provincetown. It has everything people and artists like: the romance of the shore, good climate and proximity to the city."

The retirement lasted about a year. "I was bored out of my mind," he says. Vicki Seitchik, Harry's mate, friend and business partner, convinced him to buy the Carroll Villa and its restaurant. Six years later, the complex on Jackson Street is a combination art gallery, cultural center, artist's community and nascent sports center (Harry wants to build a handball court this winter.. His staff of artists and chefs schooled in Paris return year after year, attracted by the creative freedom Harry fosters.

Wasting no time after the greeting, we sat down to eat. Harry joined us for a moment to tell a few stories. The first was about his staff, his "waitstaff."

"A waitress you pinch," he explains. "Waitresses just carry food, but they have no involvement with it. Here the whole process of making and serving flows together. When I say waitress, it's an insult." A lot of the waitstaff has been with the Batter for five years or more, and many of them are photographers or painters.

"I keep trying to get out of the art business," says Harry with hardly a grain of conviction, "but it keeps coming back." No wonder. The main dining room doubles as an art gallery. On this Sunday watercolors by local artist Peggy Scheper are on display. Five sold in the first weekend.

"Something is always going on here," says Harry, pointing to a yellow 1968 Mustang "wrapped" in red crepe ribbons and parked in the driveway next to the hotel. "It's a surprise present for a sweet-sixteen party out back," said Harry. He left us to our brunch decisions.

Would it be the whole-wheat pancakes with creme anglais listed on the blackboard, the Poisson Fume Nicoise (local fish smoked at the Mad Batter, blended with celery, red onion and a light mustard dressing, served with new potatoes, olives, eggs and beans, poached with herbs and olive oil), or maybe the Three Delights Crepes (shrimp, scallops and chicken in a sauteed mushroom sauce)?

I went for a brunch mainstay, eggs benedict, and Marsha had the omelette d'amour (smoked oysters, shallots and mushrooms sauteed in red wine and topped with sour cream and chives). Then we waited. Waited and talked. Food takes a long time at the Mad Batter, but Harry has it figured out.

"People who wonder where the food is and complain about slow service are not very compatible," he explained to me. "Figure it this way. If you're at the table with someone you like, you sit back and relax, share some wine and conversation, gaze in one another's eyes.

"But if you're bored with the person you're with," he said, "you start looking around for someone to take your boredom out on, like the waitstaff. We don't pretend to be a fast-food restaurant. We prepare everything to order and that takes time. Slowness is our trademark. The problem is, how do we attract more lovers to our restaurant?"

Philosophy aside, I was happy when brunch arrived. We topped it off with homemade peach sorbet, then Harry shooed us to the beach.

The Carroll Villa sits on a small street two blocks from the beach, just past a miniature golf course and a pizza joint. Like most East Coast beach resorts on a sunny summer Sunday, Cape May beach near the center of town is a riot of colorful umbrellas, sunbathers and purple-lipped kids splashing in the surf. There are exceptions.

Take beach bocci, for example. It's the old Italian game, played on hard sand with softball-sized sponge balls rather than regulation bocci balls. We passed three games until we encountered Cape May's version of horseshoe tossing, which I dubbed clam-shoes. The game is played with Cape May regulation clam shells, about three inches across. Instead of tossing horseshoes at a metal rod, you fling the clam shells at holes in the sand about 20 feet apart.

"It's cheap and fun," said one toothless gent, wearing a felt baseball cap with no letters. "We can get new free equipment every day."

On down the beach heading west to the lighthouse, the throng begins to thin out. Every quarter mile or so, a jetty of huge black boulders juts into the ocean for a good 50 yards, and surf-casting fishermen pull in the occasional weakfish, flounder or bluefish. You can rent rod, reel and tackle for about $14 a day by the fish boats.

From the tip of the third jetty, you can see a mile down the coast to the faint outlines of an old convent and the lighthouse beyond. In between, the beach is virtually empty, except for a few strollers, even on mid-season weekends. The sand is draped on sweeping arcs from the jetties to the dunes behind. You can walk from town to the lonely beaches or drive or bicycle to the lighthouse and head back towards town. But take heed: there's no shelter.

Rather than walk up the shorelines, we doubled back to town for a tour of the historic streets. Cape May is so historic, the whole town in on the National Historic Register.

Cape May was officially discovered by Sir Henry Hudson on Aug. 28, 1609, to be exact. Now the streets are lined with huge old mansions and an assortment of architecture that survived Cape May's frequent rising and falling fortunes in the resort vogue: homes of Victorian beer barons with painted wooden ruffles and lacey furbelows; huge, pre-Civil War wooden hotels with three-story wraparound porticos modeled on southern plantation buildings; squat, bulging bunaglow-style guesthouses, vintage 1900, that seem to be mostly roof; and a couple of monstrous brick hotels, like the Christian Admiral, owned by an ultraconservative evangelical minister, the Rev. Carl McIntire.

Unlike its northern neighbor resorts, Cape May is short on plastic, glass and street motels and beach honky-tonk. The cafe-lined "Victorian" mall in the center of town even comes off with a bit of quiet allure.

And while many of the inns seem to be trying to out-Victorian one another, with brochures explaining who slept where and even a few paid tours, Harry has taken a gradual, unassuming approach to remodeling the Carroll Villa and its 28 rooms. Room by room, bathroom by bathroom, Vicki helps with the decor and Harry poses as wallpaperer, plumber and carpenter.

"We bought the inn from a fellow I used to play poker with," says Harry on a quick tour. "It used to be painted three basic colors, deep burgundy, deep turquoise and deep forest green. It looked like a bordello, but I guess those were real Victorian colors."

He repainted. The result is understated charm. Most of the rooms are spare, starched and simple, rather than frilly. No phones. No TV. The rooms without private baths have a small sink hung on the wall like a piece of sculpture. Room 28, our favorite, has an antique bed with a solid walnut headboard that's so tall it almost reaches the ceiling, lace curtains billowing with an ocean breeze, a vase of dried purple flowers and a rocking chair.

By about 6 o'clock in the evening, catching the ferry back to Delaware had slipped my mind, or at least I ignored it. I wandered into the kitchen at the Mad Batter. Dinner chef Mindy Silver was arranging six steamed clams in a bowl of red sauce with basil and fennel and sprinkling them with fresh parsley for Zuppa de Clams. She was wearing a white apron over her sleeveless T-shirt, had three earrings in her right ear, and was glistening from the steamy kitchen.

Mindy and the two dinner cooks, Joe and George, have worked together at the Batter for five years. It's become a community, she said. "We have a lot of creative freedom to change the menu as soon as we get bored," said Mindy, twirling a limp strand of pasta, which, incidentally, is one of the few foods not made from scratch at the Batter.

At Mindy's suggestion, we tried the swordfish with green peppercorn cream sauce for dinner. "I make it with creme fraiche, a combination of fresh cream and buttermilk that I culture here," she said. "It's something I picked up in Paris." We weren't disappointed by the swordfish, or the szechuan ginger shrimp, or the white chocolate mousse cake, one of the more conservative desserts.

Nightlife in Cape May is almost as Victorian as the architecture. A couple of pubs on the mall have live music, from jazz to rock 'n' roll. If your yen is for bright lights, steamy bars and traffic, just head 15 minutes up the road to Wildwood and its 1950s-vintage boardwalk.

If Wildwood is too tame, Atlantic City is an hour up the coast. But on this night, we had a ferry to catch.

We finished dinner, topped off with espresso, said our goodbyes to Harry and Vickie, and headed for the ferry about 8:15. Harry said we wouldn't make it; I said bah.

Ferries are pretty sights, I learned, when they pull away from shore and the horn sounds so loud it's almost as though you were on board. We weren't.