The Prince George's County Council, seeking ways to deal with the shrinking capacity of its garbage dumps, voted yesterday to prohibit the dumping of trash from outside Prince George's in county-owned landfills.

The council also rejected a proposal to expand by 103 acres the Sandy Hill Sanitary Landfill near Bowie after several area residents complained about the plan. That landfill, which was to be in operation until 1993, is now expected to reach capacity within the next five years.

Council members said it was necessary to prohibit the dumping of trash from outside the county in order to extend the life of Sandy Hill, which serves the densely populated northern end of the county. Sandy Hill receives about 1,100 tons of solid waste daily, much more than the 6,000 tons a week originally projected.

About 15 percent of the trash dumped at Sandy Hill comes from outside the county, much of that from Montgomery County, according to Hugh Robey of the Maryland-National Park and Planning Commission, which owns Sandy Hill.

Robey said Sandy Hill is attractive to trash haulers who do business in Montgomery because Prince George's charges $20 for each ton of refuse dumped, compared with the $32 a ton charged at Montgomery's Laytonsville landfill.

John Menke, Montgomery's director of environmental services, said he did not think the new Prince George's law would pose a a hardship to Montgomery, but added that he thought such a measure would be difficult, if not impossible, to enforce since a single hauler may be carrying trash from several different places.

It was the second time the Prince George's council has attempted to restrict dumping in the county. The last time, the federal court in Baltimore ruled that such a measure interfered with interstate trade.

The law passed yesterday is more narrowly written in an effort to overcome legal difficulties and applies only to county-owned landfills.

The measure was opposed by the Maryland Solid Waste Association, which represents trash haulers. Kathy Marks, that group's executive director, told council members the law would subject the county to "massive, expensive lawsuits."

The council also heard from a string of Bowie area residents, including two state legislators, who opposed any expansion of the Sandy Hill dump and urged finding alternatives to deal with the trash-disposal problem.

The proposal to expand Sandy Hill was rejected by a 6-to-2 vote, with council members William Amonett and Hilda Pemberton voting in favor of the expansion; member James Herl was absent.

For the past year the council has been studying a plan for disposing of solid waste through a burning process that would generate steam heat for buildings. The council has attempted to locate possible sites for steam-generating facilities at the University of Maryland, according to Dennis Bigley, a management specialist for the council. But Bigley said it would probably take seven years before such facilities could be put to use and a landfill would still be needed for that portion of solid waste that cannot be burned.

In other action, the council approved the acquisition of 124.6 acres adjacent to the Brown Station Road Landfill outside Upper Marlboro to serve as a buffer zone between the dump and residential areas.