In return for the vote of Rep. Edwin B. Forsythe (R-N.J.) in Thursday's veto battle in the House, the White House agreed to issue controversial regulations barring Japanese tuna fishermen from certain U.S. Atlantic coastal waters.

The decision, according to administration sources, was made by Office of Management and Budget Director David A. Stockman at the urging of House Minority Leader Robert H. Michel (R-Ill.), who called him in Forsythe's behalf.

As a result, Forsythe reversed his earlier position on the $14.1 billion supplemental appropriations bill and voted to sustain President Reagan's veto. The administration, for its part, immediately approved the regulations and sent them to the Federal Register for publication next week.

Forsythe acknowledged yesterday that his vote was linked to his longstanding interest in winning support of the fishing prohibition. In August Forsythe had voted against the president on the supplemental money bill.

He said his objective was protection of American sport fishermen, who complain increasingly that their gear is damaged and snarled in 50-mile lines strung out by Japanese vessels in the area.

The regulations, drawn up by the Department of Commerce's National Oceanic and Atmospheric Administration, will take force 10 days after publication. In effect, they will preclude Japanese tuna fishing in U.S. waters north of Cape Lookout, N.C.

"This was an extremely painful decision for the administration," said one official. "It could have ramifications for U.S.-Japanese trade. It might violate other voluntary fishing agreements we have with the Japanese, for example in the Gulf of Mexico."

The regulations have been debated for months inside the administration. Supporters of regulatory reform have opposed their issuance; officials worried about possible Japanese trade retaliation opposed them; National Marine Fisheries Service vigorously supported them.

The bureaucrats were spurred by Forsythe, the ranking Republican on the House subcommittee on fisheries and wildlife conservation, and Rep. John B. Breaux (D-La.), the chairman, who threatened legislative action against the Japanese unless Commerce moved with its regulations.

Michel's office confirmed that the GOP leader had phoned Stockman from the House floor "to get a decision on those regulations" after Forsythe indicated his vote might be available to help sustain Reagan's veto. OMB spokesman Edwin L. Dale Jr. refused to comment on the vote-trading reports and Stockman's role in the matter, but he said the fishing decision "was not a major issue for us."

The regulations, according to Commerce, are intended "to make available additional billfishes and sharks for domestic fishermen and to reduce gear conflicts between U.S. and foreign fishermen."

To achieve this, they will close the area north of Cape Lookout to foreign longline operations, that is, about a dozen Japanese fishing vessels that operate there; impose fines up to $500 per billfish and $272 for each swordfish illegally pulled up; ban foreign fishermen from areas in which U.S. operators have placed "fixed" gear.

A problem with the Japanese long-line tuna fishing techniques is that they regularly catch billfish that must, under U.S. regulations, be returned to the sea. U.S. commercial and sport fishermen contend that as many as 70 percent of these billfish die after they are thrown back, adversely affecting American fishing.

Commerce officials say that the bluefin tunas sought by the Japanese are found in greatest abundance in the same areas favored by billfish, which are pursued by U.S. fishermen. As one result, foreign-domestic conflicts arise over tangled gear.

The Commerce regulations, in addition to closing the upper Atlantic stretch immediately, will affect the area south of Cape Lookout from June 1 through Sept. 30 and another area in the Gulf of Mexico, not now fished by the Japanese, from May 1 through Dec. 31.

Spokesmen for the Japan Fisheries Association, which has opposed issuance of the regulations as a violation of U.S. law that establishes fishing management zones, were uncertain yesterday if if the organization would attempt to block the new restrictions through litigation.