Factory orders jumped by a strong 3.2 percent in March, providing another sign that manufacturers are joining consumers in the economic recovery, the Commerce Department reported yesterday.

The increase to $161.8 billion--spurred by a sharp rise in orders for office and computing equipment--was the fourth in five months, and the dollar total was the highest since the summer of 1981.

Shipments also rose, by 2.4 percent to $160.2 billion, the report said.

On the other hand, manufacturers' inventories declined by a huge 1 1/2 percent for the 13th straight monthly decrease, indicating that factory owners were hardly rushing to replenish stockpiles by actually increasing production.

Commerce Secretary Malcolm Baldrige said yesterday that the declining inventories and rising shipments and orders means that "future demand gains will have to be matched by further increases in production."

Otto Eckstein, chairman of Data Resources, said the March orders mean "business is beginning to join the recovery."

"Consumer confidence jumped in March, housing sales rose, car sales went up in April. Now you have a big jump in factory orders, the single most significant indicator of business and a pivotal part of the recovery," Eckstein said. "It looks like the domestic economy is really going."

In all, orders totaled $161.8 billion in March, the highest level since September 1981. The figure is adjusted to discount for normal seasonal variations, but not for inflation.

March orders for durable goods rose by 3 1/2 percent to $80.8 billion, rather than by 0.3 percent as the Commerce Department had estimated less than two weeks ago. The unusually large revision apparently occurred because a lot of orders showed up in late responses to the department's durables survey.

Within the durable-goods category, orders for nondefense capital goods rose 10.7 percent to $20.8 billion for the largest one-month increase for that important category of business equipment in two years, and new orders for machinery increased 11 percent, the largest gain for that category since January 1980. Most of the gain in both categories "was attributable to the office and computing equipment industry," the report said.

New orders for military hardware rose 25 percent, reversing the large declines for that volatile category that were registered in January and February.

March orders for nondurable goods rose 3 percent to $81 billion, the report said.

Total inventories dropped to $257.8 billion. Except for a small increase in February 1982, inventories have been declining since November 1981, when they were estimated at $285.8 billion.

With shipments rising and inventories declining, the inventory-to-shipments ratio dropped from 1.67 in February to 1.61 in March, the lowest level since June 1981.

Meanwhile, the F.W. Dodge unit of McGraw-Hill Inc. said yesterday that the value of new construction contracts climbed 10 percent in March to $16.2 billion, as strength in homebuilding and public works construction carried the recovery of the construction industry another step forward.

And the Commerce Department has reported that the per capita personal income of Americans rose slightly less than their cost of living last year, going up 5.3 percent to $11,056.

Per capita personal income figures for 1982 for this area, accompanied by the percentage of change from 1981 were District of Columbia, $14,347, 6.2 percent; Maryland, $12,194, 6 1/2 percent; and Virginia, $11,003, 6.2 percent.