The United States and Cuba made substantial progress toward an agreement on fisheries and maritime boundaries in New York talks during the past week, the State Department said yesterday.

Spokesman Frederick Z. Brown, amplifying a joint communique issued by the two delegations, said the purpose of the next round of talks - for which no date was announced - will be to conclude a fisheries-maritime agreement.

U.S. sources said the two sides were very close to agreement when the talks ended late Tuesday, but added that highly technical matters are involved.

Brown said the discussions between eight-member teams of officials dealt exclusively with fisheries and maritime boundaries. President Carter and Secretary of State Cyrus R. Vance have expressed U.S. willingness to discuss a broad range of issues in an effort to improve relations. However, the Cubans declined to engage in broader discussions now, according to informed sources.

A U.S.-Cuba anti-hijacking agreement, which was renounced by President Fidel Castro last October, is due to expire April 15. There is no indication of Cuban willingness to negotiate an extension of this pact before it expires, but officials held out some hope for a Cuban decision to extend it by unilateral action. The United State favors continuation of the agreement.