David Treen, the first Republican to be elected governor of Louisiana in more than a century, yesterday pledged to make election reform his first priority in office as he watched his victory margin in Saturday's election shrink amid some reports of vote fraud.

With a narrow but apparently comfortable lead early yesterday, Treen, a four-term congressman, cautiously claimed victory over Democrat Louis Lambert, a self-styled populist.

"As we have the right to be proud, let us be prudent," Tren told a crowd of cheering supporters at a hotel in New Orleans' French Quarter. "We have a very, very narrow margin of a little over 17,000 votes. So it behooves us to be prudent in our claims. If the official results, of course, confirm this, we will have won."

By yesterday afternoon that margin had dropped by 7,079 voted, apparently the result of reporting errors in the unoficial taly of the runoff election. Before the retally, with all of the state's precincts reporting, Treen had led Lambert, chairman of the state Public Service Commission, by 693,122 votes to 675,579, or 50.6 percent of Lambert's 49.4 percent.

Fearing that Democratic election officials might tamper with voting machines Treen dispatched security guards to voting places and obtained a court order impounding the state's 6,400 voting machines pending confirmation of his victory.

"We're nervous. We're very nervous," said John Cade, Treen's campaign manager. "It's a situation where the establishment is against us and the establishment holds the keys."

Cade's fears are not without foundation. Louisiana has a long tradition of tolerating vote fraud and Saturday there were numerous reports of intimidation, harassment and vote-buying at the polls in New Orleans and the southern tier of the state's parishes.

If Treen's win holds up, he will become the first Republican to sit in the Louisiana governor's chair sinch William Pitt Kellogg, a carpetbagger from Vermont who served during the Reconstruction years of 1873 to 1877.

Treen's election is a reat personal triumph for the 51-year-old conservative Republican and for Louisiana's small Republican Party, which has only 5 percent of the State's registered voters.

Treen's victory puts into Republican hands one of the nation's most powerful gubernatorial offices, one with unusually strong appointive and patronage authority. This could prove important in the 1980 presidentail race. Carter narrowly carried Louisiana in 1976, but Richard Nixon won it in 1972 and former Alabama governor George Wallace, running as a third-party candidate, won it in 1968.

Treen won with the support of much of the state's Democratic candidates of governor. He seldom mentioned his Republicanism in his campaign literature and advertisements. His efforts to avoid the party label were aided by an unusual election system in use this year for the first time.

Under the system, candidates from both parties run in a nonpartisan primary. If no candidate obtains more than 50 percent of the vote -- which was what happened in the state primary last Oct. 27 -- a runoff election is scheduled between the two leading contenders.

The results of Saturday's runoff suggest that the voters split pretty much along racial and economic lines. Lambert, who campaigned with the support of organized labor, endorsment from outgoing Gov. Edwin Edwards, and the backing of most of the state's black leaders, did well among blacks and lower-income whites and in Louisiana's Cajun areas. Treen scored heavily in white suburban areas and the northern part of the state.