The White House has put off unveiling its 1981 farm bill, apparently to keep from offending members of Congress whose support it needs on dairy price support votes this week.

Administration sources confirmed that concern over the dairy votes was a factor in yesterday's abrupt cancellation of plans to announce details of the farm measure.

Agriculture Secretary John R. Block was to have appeared today before the House Agriculture Committee, and tomorrow in the Senate, with the overall farm package, which sources said would include proposed longrange cuts in the dairy program. These would come on top of the diary cut now at issue: cancellation of an April 1 price support increase.

Block's office insisted yesterday that cancellation of the secretary's announcement of the farm proposals was necessary because final details had not been worked out.

But other sources said a factor was the Senate vote set for this afternoon on the bill dealing with the April 1 increase.

The House is expected to take up the diary bill Thursday. Passage is anticipated in both chambers, but a Republican leadership spokesman said, "The timing of Block's testimony up here would be deleterious to the dairy bill. . . . We want to win the dairy bill."

Block has said that he wants to propose modest price support increases for other commodities covered by the farm bill.But Republicans on Capitol Hill were concerned that the dairy legislation might be undercut if Block, in disclosing his long-range plans, talked about more cutbacks.

More cuts in the diary program apparently are in the works, although an administration source said yesterday that details still are being worked out in negotiations between Block and David A. Stockman, director of the Office of Management and Budget