The Reagan transition team, in a brief filed in U.S. District Court here, has strongly hinted it may seek to withdraw a sweeping affirmative action agreement guaranteeing more federal jobs for blacks and Hispanics.

The Reagan brief raises the question of whether the agreement undermines the merit principle in federal employment and sets up an impermissible quota system to fill some of the better entry-level jobs at grades 5 and 7.

The agreement takes the form of a legal consent decree to settle a minority lawsuit against the government. It was signed Jan. 9 by Associate Attorney General John H. Shenefield on his last day in office and submitted to Judge Joyce Hens Green, despite a request from the Reagan transition team that Shenefield leave the matter for the new administration to decide.

The agreement received preliminary approval from Green Thursday but won't become effective until she gives final approval.

Under its terms, the government over the next three years would phase out the Professional and Administrative Career Examination (PACE), the most important civil service general competency test, which has been used to fill 118 categories of federal jobs, because of charges that too few blacks and Hispanics pass and get jobs.

In its place, agencies would construct new tests designed both to find competent people and to guarantee that a much higher percentage of black and Hispanic candidates ends up passing and getting jobs.

The new tests would seek to assure that the number of blacks and Hispanics who pass and get jobs would be roughly proportional to the number who took the tests. thus, if half of those taking a test for a specific job were blacks or Hispanics, then about half of those passing and ending up with jobs (but no less than 40 percent) would have to be blacks or Hispanics. If the new tests didn't produce this result, then agencies in some cases might be required to junk them and construct still other tests.

In addition, federal agencies would have to make special efforts to recruit black and Hispanic employes, sometimes hiring them on the basis of academic record without a test, until blacks and Hispanics in each category of jobs reached 20 percent.

The proposed consent decree arose from a lawsuit brought Jan. 29, 1979, by a group of blacks and Hispanics who had failed to pass the April 1978 exam. Figures showed that 42 percent of whites taking the test got a passing grade of 70 or more, but only 5 percent of the blacks and 13 percent of the Hispanics. It was alleged that the tests contained hidden cultural biases and demanded knowledge not required for good performance in the 118 categories of jobs.

The Office of Personnel Management, however, has asserted that the PACE test was designed with job-relatedness in mind.

In a brief filed with Judge Green last week, the Reagan transition team asserted that there are a number of serious unanswered questions about the settlement and asked her to hold up even preliminary approval.

Among the questions: whether the requirement that special efforts at hiring blacks and Hispanics continue until they make up 20 percent of the labor force is an improper quota; whether minority applicants failed PACE for reasons unrelated to minority status (for examle, did better minority college graduates seek jobs elsewhere, leaving only the poorer ones to take the federal exam?); whether there is real evidence that PACE is faulty in measuring job-related abilities, as the plaintiffs asserted.

Despite the Reagan team's request for delay so the new administration could study the proposed consent decree carefully, Green on Thursday granted preliminary approval, saying that after intense review, she found the agreement "fair, just, equitable and appropriate under the law."

There will now be a period for submission of comment by the interested parties, after which Green will make a decision. The Reagan team will have some time to study the decree further and then submit its final views. But several attorneys said it's unclear whether, now that the matter has been placed in Green's hands, the new administration has the legal right to rescind the agreement unilaterally if it decides it doesn't like it. By speeding the agreement to court just before leaving office, Shenefield may have foreclosed the new administration's right to modify or withdraw the agreement unless Green agrees.