The Reagan administration backed away yesterday from plans to send more than 2,000 Cuban and Haitian refugees to an abandoned U.S. Navy training center in northeast Maryland after learning that the badly dilapidated base probably could not be revamped for less than $40 million.

"The cost apparently makes it somewhat unlikely that it will be the final choice," and administration spokesman said yesterday. The World War II-era Bainbridge Naval Training Center in Port Deposit was originally the administration's favored site for a long-term national detention center for illegal aliens, according to the spokesman.

The center would eventually house about 850 Cuban refugees now at Fort Chaffee, Ark., and 1,426 Haitians now in Miami's crowded Camp Krome Detention Center, according to Verne Jervis of the U.S. Immigration and Naturalization Service. It also would be used as a national processing point for aliens and refugees from across the country.

White House spokesmen said no decision on the center site is expected before Thursday. They stressed that Bainbridge is still under consideration, although no longer the front-runner.

The shift in the administration's position was greeted with delight by several Maryland politicians who have bitterly protested the plans for the Bainbridge center since they surfaced last week. They had accused the White House of wasting money and "playing political hardball" in initially planning to move the refugees to Maryland, one of only five states carried by former President Carter last November.

"Unless they wanted to play hardballpolitics and disregard all costs, it would be an absurd place to put it," said Rep. Clarence Long, a Baltimore County Democrat who is the dean of the Maryland congressional delegation and who was among the loudest protesters of the plan. "You have to see it [Bainbridge] to believe it. It's unbelievable, the condition it's in."

Long's administrative assistant, Tim Kernan, said he fell through the porch of a dormitory at the center after stepping on a rotten board during a tour on Sunday with Long and Rep. Royden P. Dyson, whose district includes the abandoned Cecil County base. "My leg fell right through.It drew blood," Kernan said, adding that he got a tetanus shot yesterday for the injury.

Initial plans to send the refugees to Bainbridge touched off a furor in tiny Port Depost (population 675) and Cecil County. County officials last week sent a task force to Fort Chaffee to learn about the Cubans, who are "mostly single men, with limited education, limited skills, almost no English ability," according to Doris Meissner, acting INS commissioner.

The group flew by helicopter to Annapolis yesterday to share its findings with Gov. Harry Hughes, who was "sore as hell" about the plans, according to an aide. "It's a helluva way to behave for an administration that talks about saving the taxpayers' money," the aide said. Hughes last week fired off a three-page telegram to Reagan expressing "my most vigorous opposition" to the plans for Bainbridge.

It was the second time in about a year that the abandoned Bainbridge center has been considered as a temporary home for refugees. Last May, the Carter administration announced plans to send some 20,000 Cubans from the "Freedom Flotilla" to Port Deposit, but canceled them after officials inspected the badly rundown facilities.

The center, abandoned six years ago by the Navy, now resembles a ghost town. Officals say the asbestos-tile apartments need extensive plumbing, roofing, electrical and landscaping repairs.