Commuter rail service linking the outer Virginia suburbs with the District of Columbia could be threatened by the announced refusal of Consolidated Rail Corp. to permit commuter trains to run on Conrail tracks, promoters of the project said yesterday.

Northern Virginia legislators and transportation planners hope to begin passenger service between Manassas and Fredericksburg and Washington's Union Station over track owned by four railroads, including Conrail.

Conrail, the freight railroad that recently was sold by the federal government to investors, owns the 1 1/2-mile stretch of tracks between the Potomac River and Virginia Avenue in the District, including the site of a proposed L'Enfant Plaza commuter rail stop. Without Conrail's track, the commuter trains could not reach Union Station and would have to terminate at Crystal City or Alexandria's Union Station, near the King Street Metro Station.

Richard B. Hasselman, Conrail's senior vice president of operations, said in a June 8 letter to the Northern Virginia Transportation Commission, "We cannot support this {commuter rail} proposal due to its serious conflict with the 14 freight and 14 Amtrak passenger trains now operating in this same area."

The commission is coordinating the commuter rail project, and envisions running four round trips a day along each corridor to Washington.

That would mean a total of 16 additional trains over the Conrail track and could disrupt freight schedules, Hasselman wrote. "We are also concerned about the safety aspects of this proposal, especially with respect to the proposed new station at L'Enfant Plaza." He did not elaborate.

Del. David G. Brickley (D-Prince William) said he wrote to Conrail yesterday to determine "what we have to do to get this resolved."

Brickley said he hoped the Conrail letter is not "the first bargaining chip" in an effort to negotiate Conrail's track-use fees.

The Northern Virginia Transportation Commission will have to pay track-use fees to the railroads involved, as any outside user would do. Conrail is known as a tough negotiator on track-use fees.

The commission is working on a plan to protect the railroads from liability in the event of an accident.

Under the proposal, the commission would buy cars and locomotives and Amtrak would operate the service under contract.

The commission recently estimated that the service would carry 4,000 riders a day, and could begin by March 1989 if all goes smoothly.