Newspaper vendors took the Metro subway system by surprise yesterday, hawking The Washington Post and The Washington Star in subway stations, selling papers aboard the trains and setting up newspaper vending boxes at subway entrances.

The two newspapers' apparent coup in what has become a year-long wrangle with Metro encountered what apparently was a setback at 4 p.m., however, D.C. Fire Marshal John P. Breen ruled that newspaper vending machines may pose a fire hazard in subway stations, and transit polic were quickly ordered to haul the offending vending boxes away.

By evening, battle lines were being drawn for a possible new match between Metro officials and the newspapers next week.

Since Washington's first subway line opened in March, 1976, Post and Star officials have sought to sell their newspapers in Metro stations. Negotiations have been under way for more than a year.

The Post and The Star contend that free press guarantees under the U.S. Constitution's First Amendment assure them the right to distribute newspapers in a reasonable manner in public places, such as subway stations.

Metro officials say that, while they hope eventually to permit Post and Star vending boxes at subway stops, more time is required to straighten out some remaining problems. They say they are seeking to make certain the newspaper vending machines do not block passageways, clash with the stations' subdued decor or pose a fire hazard.

In addition, Metro officials argue, they are worried that allowing newspaper vending machines in the stations may clear the way for other commercial enterprises st subway stops, such as hotdog stands, sales of pornographic publications or flower carts.

Fire Marshal Breen expressed similar concern. He noted that an agreement between fire authorities and Metro bars combustible material from the stations unless adequate sprinkler or other fire-suppression systems are installed. He said he fears allowing Post and Star vending boxes in stations would quickly lead to permitting other commercial enterprises at subway stops, posing a more serious fire threat.

Last month, according to Stuart F. Pierson, a lawyer representing both The Star and The Post, the two newspapers gave Metro officials a 15-day period to complete negotiations in the dispute. The deadline passed on Thursday without an accord, Pierson said, and the newspapers decided to take action.

The Post and The Star each set up one vending box at a subway stop. The newspapers sent hawkers to other stations.

John E. Warrington, Metro's marketing director, objected to the moves, saying, "All of a sudden, we find that confrontation has been thrust upon us." Lawyers for the newspapers said last night that newspaper vendors would return to the subway stations Monday.