A U.S. appeals court yesterday delivered a major setback to Virginia's tobacco and apple growers by ordering them to pay imported foreign laborers a wage higher than Americans earns for the same work.

"This decision is just devastating to the tobacco growers," said Albert Misler, an attorney for Virginia's major producers of tobacco who last year harvested crops valued at $167 million and filed the suit decided yesterday. "It gives the Labor Department a free hand and it's a rude shock to the farmers."

The 4th U.S. Circuit Court of Appeals overturned a lower court decision that exempted Virginia growers last year from paying about $1 million to alien tobacco workers alone -- mostly Mexicans and Jamaicans imported to pick crops. U.S. District Judge James C. Turk of Roanoke had ruled that former Labor Secretary F. Ray Marshall acted "arbitrarily and capriciously" in setting the higher wage, which is designed to stem the rising influx of foreign workers.

It is unclear whether apple and tobacco growers will be forced to pay the extra money for work performed by the aliens last year.

S. Steven Karalekas, an attorney for the East Coast Apple Growers, said he believed the impact of yesterday's decision might be muted by the greater willingness of the Reagan administration to consider the growers' viewpoint.

"I'm not sure what impact this decision will have," sid Karalekas, who said Virginia apple growers last year imported 1,000 Jamaicans to pick crops in the Shenandoah Valley. "But I think the judgment of the [Reagan] Labor Department is going to be that growers should not be penalized because they can't get U.S. workers. I think the rules of the game are going to be changing in the future."

About 1,100 aliens included in a tobacco work force of 19,000, were paid $3.20 an hour rather than the $3.51 rate set by the Labor Department. The wages paid the alien apple pickers varied.

The appeals court decision comes just days before the Reagan administration formally dismantles one of the Marshall's last official acts. Last January, in the waning days of the Carter administration, Marshall ordered growers in 14 states with high concentrations of foreign workers to pay them a standard hourly wage in 1981 of $4.51, more than $1 per hour more than Virginia workers would receive.

The nation's growers lobbied strenuously against the Marshall proposal, claiming they were being unfairly penalized because Americans refused to perform the difficult physical labor. Labor Department officials said yesterday that they instead plan to continue a practice begun in 1968 of computing different hourly wages for foreign and domestic workers in each state, based on economic conditions in those states.