When the contingent of 300-plus mentally retarded youngsters, along with their chaperones and local sports and political celebrities, exited through the iron gates of the Lorton Reformatory yesterday at the end of the Special Olympics program, the man who make it possible stayed behind.

It will be at least 12 years before Sidney Davis takes that same walk because he is serving a minimum 20-years sentence for first degree murder.

But that is of little importance to the children who have participated in four Special Olympic events at Lorton; or to the other inmates who play in the intramural basketball league Davis organized, or to Washington businessman Joe Wheeler, who says his outlook on life has been influenced by the 32-year-old Davis.

Wheeler, who has been involved in attempts to get a baseball team here, first met Davis on a visit to the prison four years ago. Disenchanted with the rehabilitation programs offered by the institution, Wheeler decided to do what he could to help some of the inmates.

"I asked three prison officials for a list of the five men they considered to be the most incorrigible," Wheeler related. "Sidney Davis" name appeared on all three lists. When I met him he obviously had a chip on his shoulder, but there was something there I couldn't put my finger on. Whatever, he was definitely a promoter." Wheeler left Davis his business card and promised to keep in touch.

That same year Davis became a born-again Christian. "I began to see reality differently," Davis recounted. "I wanted to help others." He had read about the Special Olympics events in the area and considered trying to hold an event at the prison.

"We (the inmates) thought we'd cornered the market on suffering," said Davis. "But here were kids even less fortunate than ourselves."

Davis contacted Wheeler about the idea. There were obvious problems. The prison administration was hesitant to mix retarded youngsters with convicted criminals. But Wheeler, who refers to himself as "Mr. Outside" and Davis ("Mr. Inside"), helped bring in the first group of youngsters the following year.

Inmates, regardless of the crime for which they had been convicted, were allowed to chaperone one youngster each for the entire day of activity. There have been no problems in the four annual events. Wheeler recounts, "When a kid fell down during a race, his chaperone would rush out, scoop him up, and run to the finish line with him in his arms."

The programs have had a great impact on Davis' evaluation of his self-worth. "I had an opportunity to utilize what I had been given," said Davis.

"I felt a transformation within myself. It gave me a sense of worth, of contributing to society. Here we were sharing love with kids less responsible than ourselves. You just can't describe the boundaries of love."

About the only problem with the Special Olympics programs at the facility are a shortage of enough kids to go around. Since the prisoners and youngsters spend the day on a one-on-one basis, inmates who have been left out have griped about that fact at past events.

"The kids have always worn blue tee-shirts and their chaperones gold ones," said Wheeler. "This year Davis persuaded me to buy 1,100 extra gold shirts so that the inmates watching the proceeding could feel like they're a part of everything."

During his incarceration Davis has completed requirements for a high school degree and will, in August, receive an associate in arts degree from the University of the District of Columbia's extension at the prison. He will next be going for a bachelor's degree in urban planning.

He has no chance of release until he completes the minimum 20-year sentence in 1990. "I feel that Sidney Davis has been psychologically rehabilitated even though he is still legally bound to the system at Lorton," commented Wheeler. "He has completely changed his outlook even though he has no hope of getting out early.

"Four years ago Sid Davis was spending every other week in the 'hole' (solitary confinement). If you asked those same prison officials today to list the five best prisoners, I have no doubt that Davis' name would appear on every list."