The Prince George's County Council voted yesterday to close the Western Branch composting facility in Upper Marlboro, citing persistent citizen complaints about the smell and the cost of running it below capacity.

Closing the expensive, new facility will require the county to return to the landspreading method to dispose of its daily share of sludge, the solid waste from treated sewage that comes from the Blue Plains treatment facility in the District. The county is responsible for disposing of 350 tons per day. Although Western Branch was designed to process up to 600 tons per day, by order of County Executive Lawrence J. Hogan it had been operating at only a sixth of its capacity since January.

Hogan recommended last week that the $6 million facility, opened last fall, be permanently closed. Hogan had shut the facility shortly after it opened because of complaints of "unbearable and sickening odors." He allowed it to reopen in January only after ordering officials to operate it well below capacity.

Council members who disagreed with closure argued that the state had not ruled on the feasibility of a total landspreading method, and that the facility had not been given enough time to work.

The 7-to-4 council vote on Western Branch must also be approved by the Montgomery County Council, which is expected to go along because the facility is located in Prince George's County.

The Prince George's Council yesterday also transmitted to Montgomery its version of the bicounty Washington Suburban Sanitary Commission 1983 budget. It trimmed a requested 15 percent rate hike to 11 percent.