By a 4-to-4 tie, the Senate Energy Committee yesterday rejected an amendment that Sen. Howard M. Metzenbaum (D-Ohio) said would "gut" a bill to overhaul procedures for awarding oil leases on the nation's Outer Continental Shelf.

Purpose of the bill is to provide more competition and otherwise see to it that the government gets its money's worth in selling drilling rights for oil and gas on submerged land owned by the United States up to 200 miles off the coast.

Under present procedures leases are awarded to the highest cash bidder. A big oil company or a group of them may get together and bid $100 million for the right to seek oil and gas in a specified acreage under the sea. The result is that relatively small operators sometimes can't compete. Leases often go to big companies which may be in no hurry to start production until prices rise.

The bill before the Senate committee would provide various alternative bidding procedures including offering royalties. This would permit a smaller operator or offer little or no cash but to agree to pay the government a specified per cent of each barrell of oil produced.

The committee staff proposed that the "variable royalty" alternative be replaced with a producer-and consumer-oriented members which resulted in the tie vote killing the amendment for now. It could be reconsidered before the committee completes action on the bill.

Metzenbaum said the present cash bonus system "plays into the hands of the big operators" and that the bill should be defeated if the royalty option were deleted.

Chairman Henry M. Jackson (D-Wash.) sided with Metzenbaum, saying. "The big mistake we have made in handling the Outer Continental Shelp leases is that the large companies preempted the bidding. The independence companies were automatically exluded. One major - Exxon - can go it alone. Even the majors have to joint venture on the big ones. There should be a joint venture with Uncle Sam.

Sens. J. Bennett Johnson Jr. (D-La.) and Dewey F. Bartlet (R-Okla.) representing oil and gas producing states, both opposed permitting an unlimited royalty procedure. They said it would bring in speculators and losers. A winning bidder might agree to so high a royalty that he couldn't make a profit and would abandon a field "prematurely," they argued. Johnson joined three Republicans in a losing effort to delete the royalty provision.

The bill would also:

Give coastal states a large say in offshore drilling policy. Governors or regional advisory boards could recommend where there should be no drilling.The Secretary of Interior could ignore them only by ruling that the national interest requires it.

Create a new oil spill liability system under which lesses where the spills occur would pay cleanup costs and the first $35 million in damages.

Require leaseholders to file exploration as well as development and production plans so the federal and state government would know what the lessees plan to do at each stage. Lessees would be required to display "due diligence" in drilling and not sit on their leases.

Empower the federal government to conduct exploratory drilling on the shelf before the land is leased so it has some idea what it's worth.