A career British diplomat, Sir Nicholas Henderson, was named late tonight to become Britain's ambassador to the United States, replacing Peter Jay.

Henderson, who has served as British ambassador to France, West Germany and Poland, had just retired last month at age 60 after 27 years in the foreign service, returning here from Paris.

The new Conservative prime minister Margaret Thatcher, asked Henderson to come out of retirement to go to Washington just a week after she had offered Britain's most prestigious diplomatic post to Conservative former prime minister Edward Health, who turned it down.

The embarrassment that ensuing publicity about Health's refusal had caused Thatcher and, presumably Jay, may have led to Henderson's hasty appointment and its unusual midnight announcement here. It was made shortly after a dinner tonight attended by Thatcher. Foreign Secretary Lord Peter Carrington, and visiting U.S. Secretary of State Cyrus Vance at the residence here of U.S. Ambassador Kingman Brewster.

Jay, who had returned to London to attend this weekhs meetings with Vance and Carrington, as it customary, formally submitted his resignation to Thatcher today, according to a foreign office spokesman. The spokesman said no particular significance should be attached to the timing.

It was known here that Jay, the son-in-law of former Labor prime minister James Callaghan, had routinely submitted his resignation when Thatcher replaced Callaghan as prime minister earlier this month. At that time, it was expected that his resignation would not be accepted immediately and that there would be no hurry in selecting his successor.

Jau, who has been reported here to be interested in staying in the United States in an academic, foundation or journalistic job, has no firm plans yet, according to sources here. Neither Jay nor Henderson could be reached for comment early this morning.

Henderaon, a university professor's son who was educated at Oxfore, served in diplomatic posts in Egypt, Greece, Austria, Chile and Spain, and as a Soviet specialist in the Foreign Office here, before becoming British ambassador to Poland in 1969, ambassador to France in 1975.

He is married to former Greek journalist Mary Cawadias and has written several magazine articles and a book on various historical topics. He is also a tennis player and an avid gardener who staged gardening exhibits at the British ambassador's residence in Paris.