Prince George's County's 1,500-member public employes union voted yesterday to end its 11-day strike despite its inability to win a contract settlement from County Executive Lawrence J. Hogan.

After an emotional, hour-long meeting, members of the striking American Federation of State, County and Municipal Employes voted by a 5-to-1 margin to return to their county jobs Monday.

The union claimed that the strike crippled administration of the county government, but county officials disputed this. The strikers -- including clerical and maintenance workers, road repair crews and landfill operators and jail guards -- make up less than half of the 3,600 county government workers. Police officers and firefighters are represented by different unions and did not strike.

The union members also voted by the same 5-to-1 margin to accept a six-point proposal to resolve their protracted labor dispute. The proposal, submitted by a federal mediator, conforms in almost all respects to county government positions.

County negotiators said earlier in the day they would accept all of the six points except one relating to security issues.

Union leaders said they urged their members to return to work because mediation sessions with the county failed to resolve the contract dispute.

Instead of continuing the strike, which forced their members to go without pay, union leaders said they will pursue a legal action before state appellate courts that would force the county to implement a tentative agreement reached by negotiators for the two sides last February but subsequently vetoed by Hogan.

"We won many battles, but we haven't won the war yet," said union representative Paul Manner. "The war is to be fought in the courts."

In July, a Prince George's Circuit Court judge ordered the county to put that agreement into effect because the judge found Hogan had improperly vetoed it for political reasons.

The county has appealed that order, which gave the locals in AFSCME a 4.7 percent cost-of-living increase for the year ending July 1, 1980, and a 5 percent increase for the current fiscal year. That agreement also included one item, a union security clause, that Hogan has said he will not approve.

The strike that began Aug. 12 climaxed 18 months of frequently bitter clashes between the union and Hogan that included repeated strike threats, court battles and several rulings that Hogan violated county labor laws.

Hogan continually declared he would not cave in to union pressure and appealed every ruling that ordered him to settle the dispute. The union accused him of pushing it into a strike to score political points for toughness in the eyes of the electorate.

After months of vituperation, the two sides found themselves deadlocked and the union members voted to walk off the job, hoping a strike would force a settlement. When the strike finally came, wage increases had long been agreed on and only non-economic issues remained in dispute.

Strikers attending yesterday's meeting at union headquarters in Forestville had mixed reactions to the decision to call off the strike.

"I stood out for two weeks for nothing as far as I'm concerned," said Suzanne Hall, a clerical worker with the county. "I'm disappointed we pounded the beat for nothing."

Other union members held a brighter view. While the strike may not have produced a contract, it did "show Hogan that he can't just walk on us," said one striking public works employe.

Hogan was attending a convention in Ocean City and was unavailable for comment.

The union leaders say they are assuming that the February tentative agreement will be implemented eventually. In addition to the wage increases -- which were not at issue in the strike -- that agreement granted the union security clause that Hogan repeatedly has said is unacceptable.

Union leaders also said they are operating under the assumption that the six-point proposal put forward by the mediator will be the "operating document" until the court cases are resolved, as one union official put it.

The five points to which both sides agreed will allow all workers to return to their jobs at their former wage rates.

The county also will be able to discipline jail guards who walked off the job. The guards' walkout at 12:01 a.m. Aug. 12 precipitated a four-hour riot by jail inmates, and Hogan initially said he would fire striking guards. The guards have been back at work since Monday as the result of a court order. Union leaders say they will contest any firings before the county Personnel Board and in the courts.

The five points also prohibit strikers from using their annual leave time to cover the days they walked the picket lines, which means that strikers have lost nine days of pay.

The one point that was suggested by mediators, accepted by the union but not by the county, would reinstate union dues checkoffs for AFSCME members under the provisions of the union's previous contract that expired over a year ago.

The provisions of that contract were in effect until the public employes went on strike last week. Hogan said after the strike began that because of the job action the previous contract, including the dues checkoff provision, was no longer in effect.

The five-point agreement did not deal with the union's demand, which Hogan has repeatedly rejected, that all employes in its jurisdiction be required to be members or pay the union a service charge. That issue, according to union and county officials, will have to be resolved in the pending court case.