On an overcast Sunday afternoon, Patricia Taylor insisted on taking her family to Highland Beach, one last time before school started, to picnic on the sandy shore of the Chesapeake Bay.

Taylor, 53, remembers making the trip to that historic African American enclave near Annapolis so many times as a child that she and her siblings occasionally grumbled, “Oh, Lord, do we have to go this weekend?”

These days, however, she longs for more company.

“I just wish more people would come and enjoy it,” she said as she minded hamburgers on a red brick grill on the small beach. “People’s lives are so different now. Everyone is so busy just trying to pay bills. There is no time to sit around anymore.”

For generations, tiny Highland Beach was a summer haven for affluent black Washingtonians seeking refuge from segregation. Now its residents are struggling to maintain its identify while young people with no memory of Jim Crow lose their connection to what made the community so special.

Taylor started going to Highland Beach more than 40 years ago, when her parents bought a house in Venice Beach, a one-street community that borders Highland Beach and shares its past as an escape from segregation’s indignities.

Highland Beach was founded first — in 1893. Charles Douglass, a son of abolitionist Frederick Douglass, bought the land from a black farmer after he was turned away from a whites-only resort in neighboring Bay Ridge. Giants of African American intellectual life spent time at Highland Beach, including Paul Robeson, Booker T. Washington, W.E.B. Du Bois, Langston Hughes and Paul Laurence Dunbar.

For decades, the town had no sewer system and the roads were unpaved. The beach was barely longer than two city blocks. But it offered something no four-star resort could:  a vacation from racism.

When Taylor’s mother, a teacher, and her father, a small-business owner, bought the family’s cream-colored Cape Cod on Chesapeake Avenue, they envisioned filling it with children and grandchildren. Taylor, a Prince George’s County property standards inspector who lives in Southern Maryland, still divvies time up at the house with her siblings and cousins.

On that recent Sunday, despite a steady light drizzle, her husband, Frank Taylor, was firmly planted at the end of Highland Beach’s narrow pier, catching perch with his daughter Calandra Taylor, 32. Calandra’s children, who normally spend their weekends back home in Upper Marlboro, playing football or attending dance classes, bounced between a playground and the water until a discovery on the beach sent them running toward their grandmother.

“It’s a fish!” Jada Weems, 7, shrieked.

Over a chorus of “Eews,” Khaleem Washington, 10, produced a dead fish draped on a stick.

“You would pick that up,” Taylor said, her eyes resting on a gaping hole next to the gills. “We’re not cooking that.”

The boy tossed the reject onto the sand. There was no one to complain. The kids had the beach to themselves, which is perhaps the most noticeable difference to old-timers.

Up until the 1970s, it was still possible for a kid to go to Highland Beach on a summer weekend and find a dozen or more friends to ride golf carts and roam the streets. The popularity of Highland Beach ebbed in the years after segregation ended and its denizens became free to go where they wanted. The Fourth of July fireworks and Labor Day festivities were suspended for lack of interest.

But starting in the early 1990s, the generations that grew up boating to the screw-pile lighthouse, netting crabs in Black Walnut Creek and learning to drive for the first time down Highland Beach’s narrow streets came back to retire.

In 2010, the median age of Highland Beach’s 96 full-time residents was 55, according to census data. Year-round residents are now a majority. And the community is slowly growing more integrated, with 19 white and five Hispanic residents making Highland Beach their home.

The once-remote location is surrounded by development on all sides, which is a mixed blessing. Conveniences such as supermarkets and banks are a short drive away. But the beach is a magnet for outsiders. Anyone is allowed to walk onto the beach, but no one can park on the streets. A guard shack by the entrance is manned on weekends to enforce the no-parking rule. In a larger sense, Highland Beach residents are protecting more than the beautiful views of the Bay Bridge.

“There is a sense of guarding the legacy,” said lifelong Highland Beach visitor-turned-resident Craig Herndon, 65, a retired Washington Post photographer. The community “is not dying, but it’s changing into something else.”

Through the decades

Highland Beach has been through transformations before. It was one of the earliest African American beach communities, along with Oak Bluffs in Martha’s Vineyard and Sag Harbor, N.Y. By the 1950s, similar communities had sprouted across the country, including Maryland’s Arundel on the Bay and Oyster Harbor, said historian Andrew W. Kahrl, author of “The Land Was Ours: African American Beaches from Jim Crow to the Sunbelt South.”

At a time when an African American president was unthinkable, Highland Beach’s concentration of teachers, doctors and judges offered a kind of alternate universe, said Donet Graves, 65, a Cleveland lawyer whose family has had a home there since the early 1900s. “When you could see something blacks were able to achieve, it was a reminder that we weren’t the negative images we saw on television,” he said.

Highland Beach incorporated in 1922 and has been run by an all-volunteer government ever since. (Or as ex-mayor Raymond L. Langston put it, “We have home rule!”)

Self-governance was key to helping Highland Beach survive the end of Jim Crow, which coincided with the boom in coastal real estate, Kahrl said. Many historically black beaches succumbed to development and the lure of newly integrated vacation destinations.

Highland Beach also survived because of its association with Frederick Douglass, Kahrl said. As his son built a house for him called Twin Oaks, the elder Douglass wrote that he was looking forward to sitting in the house’s tower, so “as a free man, I could look across the bay to the [Eastern Shore] land where I was born a slave.”

Douglass died before Twin Oaks was finished, but the house made it onto the National Register of Historic Places and is now a museum. It remains an anchor for Highland Beach. Some of the younger black beach communities didn’t have that sort of legacy to fall back on, Karhl said.

Jean Langston, Raymond Langston’s wife, is director of the museum at Twin Oaks, which is convenient because the Langstons live next door. Their home comes with its own bit of history: It was built by Raymond Lang­ston’s grandmother, civil rights activist Mary Church Terrell and her husband, Judge Robert Terrell.

In the 1990s, Langston was part of a movement of longtime summer residents to settle in Highland Beach full time, something that wasn’t possible until sewer lines were put in in the 1980s.

“The more you travel, you begin to realize how special this place is,” Raymond Langston said. “God forbid we lose it, because we’ll never get it back.”

Langston estimates that half of the roughly 109 homes remain in families of the original owners or their friends. But with each passing year, that number edges down, for a variety of reasons. The biggest is cost.

Patricia Evans sold her family’s home in Highland Beach because she couldn’t afford to maintain it, on top of the mortgage on her primary residence and private school tuition. “Something had to give,” she said.

But others are rediscovering the pleasures of Highland Beach. Don Graves Jr., 42, spent his summers in Highland Beach as a child, crabbing and biking with friends. He started going less often in his 20s and early 30s, but in recent years he has been going nearly every other weekend for a quick escape from his job as executive director of President Obama’s Council on Jobs and Competitiveness.

Unlike his father, Donet, the younger Graves said, he did not experience Highland Beach as a kind of alternate racial universe. He said that around his children, he and his wife play down the idea that Highland Beach is special because of its demographics.

“We want them to feel like it’s an everyday part of life,” he said, “and not an experience out of the norm — people of color having a place like Highland Beach.”

He is aware of concerns about Highland Beach’s losing what makes it distinctive. But the reasons people go there are basically the same, he said: “the friendliness of the place, the fact we all look out for one another.”

End of the day

As the summer afternoon wore on, Patricia Taylor began packing up. She and her family had chosen to stay only for the day.

The grandchildren and their three friends were on the pier, where Frank and Calandra Taylor, who works at the Center for Scientific Review of the National Institutes of Health, were still fishing. As Calandra threaded a worm on a hook for Jada, she said she had not been to Highland Beach in about five years. She usually goes to Ocean City.

Jada felt her fishing line go taut. At the end of Jada’s line appeared a wriggling silver fish, no more than six inches long. Calandra Taylor held it tightly up to her ear and then to Jada’s. “It’s a croaker,” she said, as the kids huddled around her. “Your hear it?”

“I hear it!” one of the girls exclaimed.

It wasn’t big enough to eat, and Taylor would later throw it back into the bay. The kids would go home to Capitol Heights and Upper Marlboro with stories of stingray sightings and croaking fish. And Patricia and Frank Taylor would drive back to Calvert County to get ready for the workweek. And wait for another August.