The State Department said yesterday it has badly underestimated the number of refugees fleeing Laos and Cambodia in recent months, and that the United States has little choice but to open its doors to 7,000 more refugees than current plans call for admitting.

Patricia Derian, assistant secretary of state for human rights and humanitarian affairs, told a House Judiciary subcommittee that there are about 5,000 refugees with no place to go, living in boats on beach camps in Thailand and Malaysia, with others arriving at a rate of between 1,000 and 1,500 per month.

Thailand and Malaysia are the non-Communist countries nearest to those where new Communist governments have replaced fallen American-backed regimes. Because of their proximity to the new states, they are where most refugee camps are found.

Ancient ethnic frictions, and the large numbers of refugees arriving there, have caused Thai and Malaysian attitudes to stiffen, and in recent months they have pushed some refugee boats back out to sea, she said.

Normally, the United States accepts only refugees who either have relatives here, or who worked for the United States previously and are now disadvantaged because of their U.S. connections. But because the situation of the boat people is so "desperate," Derian said, State is planning to take them in as a humanitarian gesture.

When State told Congress in August that the admission of 15,000 refugees would eliminate the emergency situation that existed then, Derian said, it estimated that 500 refugees escaped each month by sea, and 1,200 per month by land.

But between August and December, she said, the actual rate was 1,000 to 1,500 monthly "boat people," or refugees who escaped by sea, and 3,000 a month by land. Although the 15,000 refugee admissions were authorized, she said, they have not made much of a dent in Thailand's refugee camp population.

The result, Derian said, has been a kind of tragedy unthinkable to many Americans. "In November, the Thais began to hold larger refugee boats off shore, and by the end of that month had forced boats carrying a total of 255 refugees back to sea."

Rep. Harold S. Sawyer (R-Mich.), responded to the State Department's plan yesterday by saying, "The federal government can force states to use a 55-mile an hour speed limit, or integrated the public schools, why can't it force some of these other countries to take refugees" by threatening to cut off funds?

The U.S. government has apparently decided that its involvement in Indochina has left some obligation to the people displaced as a result. In 1975, 145,000 refugees were taken in here, mostly Vietnamese. Last August, another 15,000 were authorized, mostly Laotians and Cambodians, and State is now planning to admit the additional 7,000 who were the subject of yesterday's hearing.

Yesterday's hearing was largely a courtesy to Congress. The final decisions on refugees are made by the attorney general, after the secretary of state has declared an emergency.

State Department officials said the plan to accept 7,000 additional boat refugees has been approved by President Carter.