The Justice Department yesterday recommended removing price controls immediately from most oil pipelines, limiting regulation to a handful of pipelines that carry refined petroleum products.

The pipeline industry has contended, and the Reagan administration has agreed, that sufficient competition exists in the industry to keep prices in check without an elaborate regulatory plan. The Justice Department seemed largely to accept that argument in a report prepared by its antitrust division.

The report concluded that only 11 pipelines, which carry refined products, and the Trans Alaska Pipeline, the nation's largest crude-oil pipeline, need continuing regulation. Further study might reduce that number, said the Justice Department, which indicated that six of the 11 eventually might prove good candidates for deregulation.

The Justice Department recommendations call for deregulating lines controlled by more than 100 pipeline companies that carry more than two-thirds of the pipeline output in the contiguous 48 states.

Oil pipelines are regulated by the Federal Energy Regulatory Commission, which issued a rule last summer saying that it would establish rates on a case-by-case basis for pipeline companies rather than setting down a general set of rules on allowable rates of return for pipeline companies.

Pat Corcoran, executive director of the Association of Oil Pipelines, said that the report "should be the starting point for further congressional hearings and dialogues on the issue" ultimately leading to deregulation.

But not everyone agreed. "We're opposed to the deregulation of those pipelines," said Bob Mullins, legislative director for the National Farmers Union. "We think it's not in the best interest of consumers, be they urban or rural. In the long run, it's probably going to be a price increase passed along to consumers."

Corcoran said that oil pipeline transportation costs are a small factor in the cost of products, accounting for only about 1 1/2 cents per gallon of the cost of gasoline. In addition, competition in the industry would hold prices down, he said.

Pipelines have been regulated since 1906 as common carriers. FERC inherited rate-making authority from the Interstate Commerce Commission.