President Reagan's new immigration policy has created the first open split between the president and his Republican ally in Texas, Gov. Bill Clements, and has failed to win notable support from the governors of the three other states that border Mexico.

In a move that has shocked the administration, Clements harshly attacked the immigration plan before a Hispanic audience earlier this month. Attorney General William French Smith and his staff have been trying since then to bring Clements back into the fold before congressional hearings on the new proposals begin.

"His views on this are very important to us and very valuable," said Associate Attorney General Rudolph Guiliani. "We thought we understood his views and that they had been factored into the final decisions the Cabinet made.

"Obviously that didn't all work out and we believe we've got to work with the governor and his staff to try to come up with something he finds sensible and satisfactory and he can support."

Clements, who considers himself a leader on immigration policy and had discussed the issue with Reagan during the campaign last year, believed his views would be reflected in the new policy. When they were not, he went before the American GI Forum annual convention and said, "I must oppose the plan if it is not modified substantially during the legislative process."

He called for changes in the "guest worker" program, in the procedures that would lead to citizenship for many undocumented workers and in the fines assessed to employers who hire these workers.

David D. Hiller, a special assistant to the attorney general, said he believed the administration's differences with Clements are not as great as they appeared from Clements' speech, and indicated no substantial modifications of the Reagan plan were being considered to accommodate the Texas governor.

Other border governors -- all Democrats -- have been less outspoken than Clements, but have hardly fallen into line behind the Reagan plan, which was released July 31.

California Gov. Edmund G. (Jerry) Brown Jr., proccupied with his attempt to eradicate the Mediterranean fruit fly, has said little about the new plan. But press secretary Cari Beauchamp said, "He doesn't like it because he thinks it won't work and it will take jobs away from Californians."

Jim West, press secretary to Arizona Gov. Bruce Babbitt, said Babbit "has real problems" with parts of the Reagan plan, including the guest worker proposal, which would provide temporary visas for 50,000 immigrants. Babbitt fears this would recreate the unpopular bracero program of the 1940s and 1950s that brought in migrant workers.

Babbitt also has problems with Reagan's proposals to levy fines against employers who hire undocumented workers. Babbitt believes that would turn employers into an arm of law enforcement, which he opposes, West said.

New Mexico Gov. Bruce King has said little about the program since it was released, but has asked a state commission to study it in advance of a meeting between the 10 U.S. and Mexican governors who share the nearly 2,000-mile border where illegal immigrants cross.

Clements' defection is by far the most serious for Reagan politically because he is the only Republican on the border and because of his help in carrying Texas last November.

The blunt-talking governor has visited Mexico nine times since his election in 1978 and considers U.S.-Mexican relations and immigration policy as areas of personal expertise.

"He'd been the one who had helped give the issue visibility," said an aide. "He'd worked with Reagan on it, the staffs had worked on it, and he just didn't think the program reflected his ideas. He wanted to send a message that people up in Washington shouldn't take him for granted." Clements "felt the plan needed to be changed and believed the best way to effect that change was to go public," said Jon Ford, the governor's press secretary. "He'd given them the benefit of his views in private and it hadn't gotten across."

Clements called the guest worker proposal, which would begin with 50,000 persons, "unrealistic and . . . doomed to failure because it failed to deal with the other 99 percent of the 3 to 6 million undocumented workers in this country."

He also attacked the administration's proposed amnesty plan as inadequate and too slow, and said the proposed penalties against employers who hire undocumented workers "are totally out of line."

"Under the Justice Departrment's proposal, obviously there will still be hundreds of thousands of illegal Mexican aliens living underground in our society," Clements said. "The illegal sub-class would continue to exist, and this is wrong.

"Further, their employers would be designated as law-breakers. My proposal would require all employers of illegal aliens to pay a relocation fee to return the workers to their home."

Ford said he did not know whether Clements had talked personally with Reagan about immigration policy since Reagan was inaugurated, but said the governor had spoken with Attorney General Smith.

Clements also phoned Smith a day or so before his speech earlier this month to indicate his disagreement, but Justice Department officials still said they were surprised by the harshness of the criticism.

Hiller said that while there were differences between Reagan's and Clements' proposals, "There is a good deal more common ground than has been reported in news accounts and I think we both recognize that . . . . There have been consultations, both between the governor and the attorney general, and between their staffs. I'm confident there is enough similarity in our approaches that we would hope the governor would support us."

The new immigration policy is likely to be a central topic of discussion when the four U.S. and six Mexican border governors meet for the second time this fall in El Paso, Tex. It was at their meeting in June 1980 that Clements advanced his own proposal, one that met with only lukewarm support.

Congressional hearings are expected to begin sometime in September.