Voters in this American territory in the Caribbean have over-whelmingly rejected a proposed constitution that would have altered the structure of all three branches of government, largely because it would have caused taxes to rise.

It would not have affected the territory's relationship with the federal government.

By 56 percent to 44 percent, voters turned down a document written last year by 60 delegates and later approved without change by President Carter and Congress.

The voter turnout was small by Virgin Islands standards. Only 39 percent of the territory's 28,000 registered voters participated in Tuesday's special referendum; over 77 percent voted in last year's gubernatorial election.

Perhaps the greatest factor in the constitution's defeat was a fear that the additional layers of local government it provided would have cost untold millions of dollars and resulted in increased taxes.

The Virgin Islands government already is strapped to find enough money to balance its budget without enacting higher taxes, cutting spending or receiving a windfall from outside the territory.

The proposed constitution would have added an extra layer of government at the island level -- administered by elected mayors on the islands of St. Thomas, St. Croix and St. John -- to carry out the day-to-day operations of such functions as garbage pick-up and police and hospital services. The territorial government now oversees these activities.

The constitution also would have altered radically the judicial setup, creating a three-member Virgin Islands Supreme Court to hear appeals of local legal matters from the existing Territorital Court. Appeals from the proposed Supreme Court would have gone directly to the U.S. Supreme Court.

Other major changes in the rejected document were for an elected rather than appointed attorney general, creation of a local elected comptroller's office and expansion of the local Senate from 15 to 17 members.

Political observers pinned the document's defeat not only on the cost factor but also on the failure of supporters to explain its provisions adequately or to conduct a get-out-the-vote drive.

Many opponents stressed that while they favor a constitution written by Virgin Islanders as a step toward greater self-government, the flaws in this constitution far outweighed its strong points.