When U.S. Customs Service officials wanted to cut back their staff in the little town of Point Roberts, Wash., this year, the decision reverberated in the halls of the House of Representatives.

In the eyes of agency officials, it made no sense to have seven inspectors so that the community could be served around the clock. Instead, Customs wanted to have only one inspector, who would work from 9 a.m. to 5 p.m. on weekdays.

But to the 400 residents of the tourist and fishing town, the Customs agents were all that protected them from the 10,000 Canadians who dropped in on nice summer weekends to enjoy sandy beaches and, in some cases, X-rated movies and the more liberal American drinking laws. The town only has two part-time sheriff's deputies.

"All we want that station there for is to control that mob, to make sure we're not letting in drugs and felons," said Scott P. Cooper, an aide to Rep. Al Swift (D-Wash.), whose district includes Point Roberts.

Last Thursday, just four days before six of the inspectors were to be transferred, Swift and Rep. Edward R. Roybal (D-Calif.), chairman of the House Appropriations subcommittee on the Treasury, persuaded Customs to delay the move for at least six weeks.

Swift's reaction was typical of many northern congressmen, upset over the agency's plans to close up to 50 of the 77 stations along the Canadian boundary and to cut the number of inspectors at 300 entry points such as airports and seaports from 4,440 to 3,750.

The stations along the northern border have 544 inspectors, and most of the 26 stations with fewer than three employes were to be closed.

Meanwhile, the Customs Service planned to beef up many of its 22 stations along the Mexican border, which already have 688 inspectors, to help wage the war on illegal drugs.

Now the House Appropriations Committee has reported a bill that would block Customs from closing the northern stations and would increase its staff of inspectors by about 200.

"It was an imprudent cut," said Aubrey A. (Tex) Gunnels, staff director of Roybal's subcommittee.

For Point Roberts, Swift was able to win a delay by making a special case: although the town is part of the United States because it lies south of the 49th parallel, it is attached by land to the Canadian province of British Columbia. Blaine, Wash., the nearest U.S. city, is five miles away across Boundary Bay.

"I just don't like the precedent that the Customs Service thinks it can forbear from enforcing the law of the land to keep our borders secure," Swift said yesterday.

He also criticized Customs for deciding to make the move without consulting the community or other agencies such as the Immigration and Naturalization Service.

"But in their defense," Swift added, "Customs is working with a sheet that is too short for the bed: both the Reagan and the Carter administration have kept the agency on a very tight budget considering the scope of their mission."

"We were not trying to save money, but to develop a better management approach," E.H. (Gene) Mach, assistant Customs commissioner for inspection and control, said of the plans to close the stations.

"Congress just has not gotten that message correctly. The proposal was not designed to weaken our borders or to harm local commerce between the two nations."