Edward Lloyd IV of Talbot County bought this house unfinished from Samuel Chase. Chase, a future signer of the Declaration of Independence, ran out of money before he could finish the building. By the early 1770s, Lloyd had made the place one of the grandest homes in Annapolis. It was the only three-story dwelling in town.
Lloyd persuaded William Buckland, who went into business for himself after being indentured to George Mason at Gunston Hall, to move to Annapolis. Buckland became overseer of the house project, then concentrated on executing the sorts of intricate carvings he practiced at Gunston Hall. In 1802 Lloyd's youngest daughter, Mary Tayloe Lloyd, married Francis Scott Key here.