When artists Harriett Sosson and Richard Samuelson finally decided three years ago that they wanted to leave Manhattan they began looking for a place in the country where they could live and make a living.

That's exactly what they found in the George Hildreth House on Jackson Street here, which they have successfully converted into a guest house.

They have joined a growing number of landlords who have moved into this quaint seaside town and helped redevelop the country's greatest concentration of Victorian architecture. The city of 5,000 permanent residents is said to have more than 600 structures from the last 20 years of the Victorian era, most of them built after a fire that swept the town in 1878.

Victorian architecture is a mix of Greek revival, Gothic, Queen Anne, Italianate and mansard, often with a lot of freelance American carpenter thrown in.

Sosson and Samuelson had been living on Manhattan's West 24th Street. They found that they were being overwhelmed by life in New York City.

"You have to work too hard just to live in the city," Samuelson said. "If you just want to go out to dinner with friends, it's a project. Monday night is his shrink's night; she goes to basket weaving Tuesday."

Also, everyone in their circle was working at jobs they didn't like - artists working as clerks, writers as cab drivers. "And we just got fed up," he said.

"The final straw came when a burglar tried to break in through the window of the loft, with me in there," Sosson said. "It was terrifying."

From time to time they visited friends who had moved to the country, as they seem to refer to anything outside New York City.

"We would go to Massachusetts or Connecticut to see them," he said. "They were all working as laborers or carpenters or something else they didn't want to do."

Then, by chance, they found Cape May, which bills itself as America's oldest seashore resort. A friend had rented a place for the summer and they came to visit three summers ago.

On their last night, they had dinner with another friend who was in the real estate business.

"He tried to convince us we should buy one of these old Victorian places here and restore it and turn in into a guest house," Sosson said. "He pointed to the house next door. We were eating at the Mad Batter (in the Carroll Villa Hotel). This place was being used as a rooming house."

It was run down and abused, she said, but they were intrigued and stayed an extra day to look at the place. They fell in love. After at first losing the house to another buyer, they finally got it - for $40,000 - and began renovating it in December 1976, to get ready for the summer business.

The house, built in 1874, is of Second Empire style made popular by Napolean III and used extensively in Paris buildings. It has mansard roofs, with slate fancy work and a lot of wooden trim, often known as gingerbread.

Harriett and Richard renamed the three-story place Poor Richard's Inn and divided it into rooms and apartments, after carving out their own living quarters. Each room was individually designed and furnished with period furniture, some of it left in the house and others found at the flea markets and auctions. Their own artwork, mostly paintings by Samuelson and collages by Sosson, and a generous display of plants complete the decor.

The Victorian houses of Cape May are mostly painted in the white and green trim style popular at many seashore resorts.

But Sosson and Samuelson decided to paint their house in the colors that were popular during the Victorian era. The end result is a eye-stopper in earth shades of brown and sand, set off by a rich red. The house is variously described by passersby as "striking," "quaint," and "a work of art." Often, Harriett says, people come to the front porch to have their picture taken.

Poor Richard's Inn has joined its neighbors on Jackson Street as a tourist attraction as well as a guest house during the season, when Cape May's population swells to 60,000. Their house is featured on a regular walking tour of the historic district of the city, along with other gingerbread castles such as the Colonial Hotel, the Victorian Mansion and the countless others along Decatur, Ocean Gureny and Columbia streets.

And Sosson and Samuelson believe they finally have arrived at the point financially where they can use the winter months to pursue their art. CAPTION: Picture 1, Harriett Sosson and Richard Samuelson have opened this guest house in Cape May, N.J.; Picture 2, a row of Victorian Homes. Photos by Charles Puffenbarger - The Washington Post; Illustration, Poor Richard's Inn