Secretary of Health, Education and Welfare Joseph A. Califano Jr. yesterday rejected a North Carolina plan for desegregation and said he is moveing to cut off $90 million in federal aid funds.

Califano said he is taking the move "as a last resort because of the failure of the state" to come up with an acceptable plan to end vestiges of segregation in the 16-unit system -- despite nine years of negotiationa and court proceedings.

North Carolina thus could become the first state barred from federal school aid by administrative action under Title VI of the 1964 Civil Rights Act, which prohibits U.S. support of segregated activities. But Califano made clear he hopes the dispute will end in a compromise before that stage is reached.

The action by Califano, who aleady is deeply resented in tobacco-growing North Carolina because of his antismoking campaign, could cause another political fiestorm in the state.

Some North Carolina democrats have said the university dispute, combined with the smiking issue, could threaten President Carter's ability to carry the state in the 1980 presidential election.

Califano was under a court order either to accept or reject the latest North Carolina plan to end remnants of segregation at the five predominantly black and 11 predominantly white campuses of the North Carolina state university system. He faced contempt charges if he failed to make a decision on the plan and then act to carry out the decision.

Under the law. funds would not be cut off immediately. Rather, a lengthy proceeding must stat before a departmental administrative judge in which both HEW and the North Carolina system argue their positions. A decision can be appealed to other bodies within the department, and North Carolina can appeal still further to the federal courts.

This proceeding could last months or even years. In the meantime the state system can continue to collect whatever money already has been voted to it by Congress under various formulas. Only i it finally loses the legal proceedings will all funds be cut off.

However, Califano said that unless a list-minute compromise is reached with North Carolina within 30 days, he will order the department to defer any action on applications for further funds for future years -- but only in a "carefully targeted and limited fashion" that probably would affect only about $10 million to $20 million out of the $90 million a year the state normally gets. These funds will be those that "might contribute to continuing segregation," such as funds used for duplicative facilities at white and black university units.

This means that Califano will chop $10 million to $20 million a year immediately out of funds for future years unti completion of the proceedings on whether all funds can be cut off. He said none of the $90 million involves student aid, which will continue separately.

Joseph L. Rauh, attorney for the NAACP and a group of parents in the North Carolina dispute, said in an interview later that Califano "did the very minimum be coud do" by othering only limited fund deferral. "This is a namby-pamby performance by Califano which invites resistance by North Carolina -- and he's gonna get it," Rauh said.

In a telephone interview from Chapel Hill, UNC system President William Friday said, "I think it's important to continue talking. He has provided a 30-day period -- we stand ready to engage in such discussions."

Last May 12 Califano jubilantly announced he had reached agreement with Gov. James Hunt and Friday on a plan to boost black enrollment at the 11 white colleges, increase black enrollment in graduate schools, equalize black and white salaries over five years and other steps.

Part of the plan called for North Carolina to study situations where a black college and wite college, in specialized programs, were duplicating each other's programs. Duplication, it was argued, didn't provide any inducement for black students to switch to white colleges or vice versa in order to get a program available nowhere else.

Last December, Califano said, the university acknowledged duplication in 58 specialized programs but declined to change it and said duplication was educationally necessary.

Califano said he'd offered to defer the requirement to end duplication for four years, until the end of the overall plan, if the university would agree to give the black schools "unique missions" and various new programs -- and the funds to carry them out -- in order to attract white students, but the state turned this down.

Califano said that at the "11 traditionally white schools only 6 percent of the full-time undergraduates are black, and at the five traditionally black schools, only 4 percent of the full-time undergraduates are white."

Califano said he had told President Carter on Saturday of his decision to move for a fund cutoff and "he feels as I do -- he wants to see this" dispute settled.

Aides said the systemhs plans to draw black students into the 11 white units of the university and to upgrade black faculty were largely acceptable, but the plans to get whites into the five blacks schools weren't and hence the initiation of the cutoff move.