A challenge to an allegedly "anticompetitive" Reagan administration regulatory action involving prescription medicines was filed in U.S. District Court yesterday by a nonprofit group.

American drug buyers annually have hundreds of millions of dollars at stake and the federal government additional tens of millions of dollars, Public Citizen said in a letter to David A. Stockman, director of the Office of Management and Budget.

Public Citizen's lawsuit grew out of a longstanding effort by major pharmaceutical manufacturers to shield the market for brand-name drugs whose patents have expired and are thus no longer protected from competition by often less costly generic versions.

For years, the Food and Drug Administration has allowed suppliers to file so-called "paper" marketing applications which assume that the drug law's requirements to demonstrate safety and effectiveness have been met by generic versions of previously patented products. This spared generic suppliers the burden of expensive retesting in animals or humans.

LAST FALL, THE FDA practice was stayed under a court order won by leading makers of brand-name drugs. But when the stay expired this month, Richard Schweiker, secretary of health and human services, directed the FDA to deny approval of pending paper marketing application's indefinitely.

Schweiker acted "illegally" and "under pressure from large drug companies," Public Citizen charged in the letter to Stockman. The group, founded by Ralph Nader and headed by Dr. Sidney M. Wolfe, also branded Schweiker's action as "a classic example of increasing the complexity of regulations to protect industry rather than the public."

In a statement the Pharmaceutical Manufacturers Association said that paper marketing applications "could prove to be an enormous disincentive to pharmaceutical research and development." Innovating firms would be deterred from publishing their findings and from developing new drugs "because second-comers would be given a 'free ride,'" the PMA contended.

According to the letter, the companies involved have drugs with patents that will run out soon. Public Citizen listed four such medicines with combined 1979 retail sales of $780 million: Valium, which ranked first in the number of prescriptions filled; Inderal, which ranked second; Lasix, sixth, and Motrin, eighth. Generic versions would cost hundreds of millions of dollars less, Public Citizen said.