Justice Department officials and Louisiana Gov. David C. Treen said yesterday that they hope a longstanding discrimination suit against the state's university system can be settled out of court.

Treen said after a meeting with Attorney General William French Smith and Secretary of Education Terrel H. Bell that the Justice Departemnt "has moved a great deal" in negotiations in the last six weeks.

The two sides are closer than ever to a settlement and will meet again next week, Treen said. But he declined to predict whether an agreement could be reached before a scheduled Sept. 14 trial.

The Justice Department filed the Louisiana suit in March, 1974, during the Nixon administration. It charged the state with deliberately segregating its college systems to keep black students separate in schools such as Grambling College and Southern University.

The suit was the first higher-education discrimination case filed by the government. Another, against the Mississippi state system, is pending, but negotitations are not under way

The Reagan administration's attitude toward such discrimination cases became apparent June 20 when Bell announced settlement of an adminstrative proceeding against the North Carolina college system. The terms included several provisions rejected repeatedly by the Carter administration.

A Justice spokesman said Bell referred to "valuable lessons" in the North Carolina settlement at yesterday's 50-minute meeting. Treen said the North Carolina settlement was "one of the encouraging things that has occurred recently. There are some elements in that settlement that Louisiana could live with."

The Republican governor said federal negotiators no longer are promoting numerical goals for white students attending predominantly black colleges in the state. The department also has dropped, as it did in the North Carolina case, insistence that some programs be eliminated at white schools to induce whites to study the same courses at black colleges.

Nathaniel Douglas, a Justice Department civil rights attorney involved in the negotiations, would not comment on the specifics. "I can say the parties are meeting regularly and have made progress in solving some of the issues. We have not resolved the whole case," he said.

Treen said a greater proportion of blacks attends college in Louisiana than in any other state. More of the state's 28,000 black students go to predominantly white colleges than to black.