Maybe the ambiance was decidely low-key, and maybe the speeches weren't exactly barn-burners. None of that seemed to matter to the 1,700 persons who gathered for scrambled eggs and spiritual uplift yesterday at Mayor Marion Barry's annual prayer breakfast, an event fast becoming a city government institution.

Barry reminded the Washington Hilton Hotel crowd that they were in the very room President Reagan was leaving two weeks ago when he was shot by a would-be assassin, and then centered his remarks on crime and violence in society.

He said he watched a recent television program in which "20 people were killed in 20 minutes," adding, "and yet we ask why it is so easy for our young people to pick up a gun and shoot somebody.

"Where are our children and what are we doing about them?" Barry asked. "Where are the parents? What about the black family? What are we going to do about the violence and insanity that pervades our society."

The breakfast, put on each year by a group of city workers known as the Mayor's Executive Fellowship, is designed as a time for bureaucrats, politicians and civic leaders to put aside their differences for a few hours and emphasize the goals they have in common.

But there was one awkward moment, when master of of ceremonies Joseph P. Yeldell -- the former head of the city's Department of Human Services, who was found innocent more than a year ago of bribery and conspiracy and now works with the city's computers -- introduced for a bow former congressman Charles Diggs -- now serving time in a Northwest Washington halfway house after his conviction on fraud and kickback charges.

The main speech of the morning was given by J. Pat Galloway, president of the Greater Washington Board of Trade, who said that the cooperation of all District residents, including the business community, will be needed in a time of shrinking government resources and diminishing services.

"Just like police officers, teachers, labor union members and ministers, business people are citizens," Galloway said. "Most are concerned, interested and involved citizens."

Galloway's low-key speech was a far cry from last year's breakfast, at which evangelist Ted Skinner emotionally challenged the city's workers to bring more compassion to their work. "A lot of times, people measure speeches by the oratory and not by the words spoken," Yeldell remarked after Galloway had finished to polite applause.

S. C. Madison of the District of Columbia House of Prayer, Lewis Anthony, of the mayor's legal staff and Del. Walter E. Fauntroy all offered prayers. Three students from the Duke Ellington School of the Arts sang two gospel numbers, while Anthony and public library director Hardy Franklin formed a duet to sing another musical selection, "What a Fellowship."