As soon as the "open convention" challenge to President Carter's nomination failed at the Democratic National Convention here tonight, Rep. Michael D. Barnes of Montgomery County, a major spokesman for the open convention movement, strolled over to the Carter campaign trailer and accepted a reconciliation kiss from the wife of Carter's campaign chairman.

For Barnes, who had leaped into the national limelight by rushing to the head of the dump Carter movement, it was the proper political thing to do, no questions asked.

But for Mary Ann Keeffe, a Kennedy delegate from the District of Columbia who had been for the Massachusetts senator since the draft-Kennedy days of months ago, Carter's victory on the open convention vote -- which was followed by Kennedy's official withdrawal from the race -- the end was difficult to accept.

"The moment you realize that it isn't going to come about is a little hard. At first it looked so good . . . But the excitement's gone out of it," a watery-eyed Keeffe said. She was oblivious to the pro-Carter hoopla being led by Mayor Marion Barry on the convention floor shortly after the open-convention issue was decided in Carter's favor.

In their own ways, the 43 Washington area delegates who supported Kennedy played out private versions of a national drama that ended tonight.

"I feel very sad and I can't say what I will do now, or whether I will support Carter," said Deborah R. Marshall, a Prince George's County Council member and a Kennedy delegate. "I have never thought of supporting Carter. I'm not going to change my mind just because we lost the battle."

For the Kennedy backers, the only major battle remaining in the final three days of the convention is a largely symbolic effort to change Carter's stance on major issues by altering some planks in the Democratic platform.

"We in fact nominated President Carter tonight," D.C. City Council member John Ray, chairman of the District's delegation and cochairman of the D.C. Kennedy campaign, said after the open convention vote. "We'll scream and yell about the platform, and then (afterwards) he'll go being Jimmy Carter again. Hopefully, someday we will have a candidate."

Carter was the overwhelming favorite among the 142 Washington area delegates who came to this convention. The Virginia delegation was divided 59-5 in Carter's favor, Maryland, 32-26, with one uncommitted. Only the District of Columbia, with 12 Kennedy delegates among its 19 members, was not with the president.

Despite rhetorical pressure from some Carter campaign leaders -- creating minor backlash in the Virginia delegation -- voting on the open convention issue was almost identical to the division in the delegations. The District voted 12- for the Kennedy position; two Carter delegates in the Virginia group voted with the Kennedy forces, and the only vote picked up by Kennedy among the Marylanders was that of the sole uncommitted delegate.

The outlook for effective reconciliation became a major theme of Democrats worried about lingering wounds of the Carter-Kennedy contest.

"We've had a good deliberative convention here, but I think we can pull together," Maryland Gov. Harry Hughes said cheerfully.He supported Carter.

So did Virginia Democratic Party Chairman Richard L. Davis, who insisted that the handful of Kennedy supporters in his delegation were "all good Democrats. I had pledged to them to work just as hard for Kennedy if he was nominated and they said they would do the same for Carter. I take them at their word."

But for many of the Kennedy delegates, including those who attempted to unseat a president from their own party and possibly mortgaged some of their political future on the Kennedy campaign, pride and principle quickly became key factors.

Rep. Barbara Mikulski of Baltimore, who was scheduled to place Kennedy's name in nomination Wednesday night, said reconciliation depended upon the "amount of warmth (President Carter) shows to Sen. Kennedy. I want to see the President treated with respect, and some very strong work on the economic platform."

After Kennedy called tonight and informed her of his plan to withdraw, Mikulski gathered the state's Maryland delegates around her and, in emotional tones, said her nominating speech "will still be in style four years from now, and I think you'll agree that our candidate will be even more in style four years from now."

D.C. Del. Walter E. Fauntroy, who supported Kennedy but has key friends among Carter's D.C. allies, conceded defeat more than an hour before the vote was taken, and said he would "campaign enthusiastically" for Carter to make sure Reagan was not elected.

D.C. delegation chairman Ray, meanwhile, squared off during one jocular but tense moment with his former political mentor, Mayor Barry, over the Kennedy allegation that Carter opposition to an open convention was an attempt to make delegates into robots.

"I lost my voice yelling, 'Free the robots. This is the biggest robot we've got'," Ray said, pointing to Barry. "Now that he's done his duty, I guess he'll vote his conscience."

"I told the mayor he's a poor excuse for a man whose reputation is based on being his own man," Ray added, as Barry smiled back.

"That's what we have after they lose, character assassination," chimed in Council member Charlene Drew Jarvis, a Carter supporter.

"No," said Barry, "this is just a friendly chat."

Jerry Oliveira of Annandale, an alternate delegate who is a member of the International Association of Machinists, which had staunchly backed Carter, said he would walk out of the convention Wednesday night to protest Carter's nomination.

"The leaders of my union's international and the members will not endorse Carter," he said flatly. Oliveira said in November he might vote for Barry Commoner of the little noticed Citizens Party or for John Anderson, the independent, "Never Carter," he said.

Some local Kennedy delegates were especially upset with Kennedy's decision against having his name placed in nomination.

"He was going to have an unemployed auto worker second his nomination, and black man from Florida," said Kweisi Mfume, a Baltimore City Council member and Kennedy delegate. "I think the nation has been denied benefit of hearing these people, and those that worked so hard on Kennedy's behalf."

Marshall, the Prince George's County Council member, said, "It seems now like there's not much reason to stay. I keep telling myself there are platform issues coming up, but I would have liked to see him come to the convention hall one more time so we could vote for the person we came all the way up here to work for."