The Reagan administration and Congress, prompted by the skyrocketing cost of unpaid, federally guaranteed college loans, are escalating their war against defaulters, this time adding a weapon aimed at the institutions these students attended.

Department officials said they will announce in coming weeks administrative changes that will eliminate from the Guaranteed Student Loan program institutions with high percentages of student defaulters. The Senate has inserted in its trade bill, now in conference with the House, a similar provision allowing guarantee agencies to bar these institutions from the program.

The moves are certain to spark controversy; loss of federal student loan funds in most cases would force an institution to close its doors.

Opponents say the tactic is unfair, punishing institutions for the sins of former students. And, they say, it ultimately will hurt future students at the institutions, particularly low-income minority students attending for-profit trade schools, community colleges and historically black institutions, where default rates tend to be highest.

But education officials and Senate supporters point to climbing costs: The cumulative figure in defaulted GSLs reached $5.6 billion last year, rising from $977 million eight years earlier. The annual cost climbed from $530 million in 1983 to a projected $1.6 billion this year, boosting the price tag for defaults to one of the Education Department's biggest spending items.

Department officials and members of Congress argue that institutions with high default rates may be exploiting students, encouraging them to attend the school with little hope of completion or job placement upon graduating. Such student loans do not become due for payment until a student leaves school.

"It really is a disgrace, schools with 50, 60, 70 percent default rates," said William Kristol, Education Department chief of staff. "Back in the late 70s, people were appalled at default rates of 10 to 12 percent. If half of the kids {with loans at an institution} are in default, something's gone terribly wrong."

The rate at which all student loans go into default has climbed only gradually in recent years. But cumulative debt has nevertheless risen dramatically, largely due to increasing tuition and the expansion in the late 1970s of the GSL program, the staple of federal assistance to middle- and lower-income students. Officials say about 3,000 of the 8,300 participating schools exceed a 20 percent default rate.

"Are we doing students a favor by letting them go to schools with very high default rates?" asked an aide to Sen. Edward M. Kennedy (D-Mass.). "The GSL program should entitle them to a postsecondary education, but not necessarily at a specific school."

The provision in the Senate trade bill, which would allow guarantee agencies to bar schools with default rates over 25 percent, is meeting resistance in the House.

"The concern is that the students who would be hurt the most are the ones who have the least access to education," said a House Education Committee staff member.

Senate supporters and education officials argue that institutions could lower their default rates through more rigid screening of applicants, better counseling and better job placement services.

"People are wondering whether institutions aren't in fact exploiting the good will of the American people or Congress in order to take advantage of some portion of the student loan program," said Bruce M. Carnes, deputy undersecretary of education.

Christopher Davis, a spokesman for the National Association of Trade and Technical Schools, said for-profit trade schools are preparing recommendations to help lower default rates, in part by ensuring that students understand they are responsible for repaying the loan.

This year a provision allowing the Internal Revenue Service to withhold defaulted funds from federal income tax returns brought in $120 million, according to a Senate aide, but did little to curb the growing cost of loan defaults.

"I'm fearful that the situation is going to become such a serious one after the first of the year that, faced with budget problems . . . and growing publicity on defaulted loans, people will reach out for the dramatic, in order to produce an immediate result," said an aide to Sen. Claiborne Pell (D-R.I.), who chairs the education subcommittee.

The aide said "quick fix" solutions could unfairly punish an entire class of an affected institution. "That will have a terrible, potential impact on young people."