The property was first settled in the late 1730s by William Prendergast, a tenant farmer who leased 200–300 acres southeast of the nascent settlement of Pawling from the Philipse family, the area's dominant landowners. He built the small house that became the kitchen wing in 1740, adding other buildings later and connecting the two through a 65-foo stone passageway.

In 1766 he became a leader in the tenant uprising known as the Dutchess County Anti-Rent War, a revolt against the quit-rents left over from the feudal system left in place by Dutch colonists in the region which made it hard for tenants to eventually purchase their land. It was quelled by troops called in from Poughkeepsie. Prendergast was tried, convicted and sentenced to hang for treason. His wife made a personal appeal to the colonial governor, Sir Harry Moore and he was spared at the last minute by a gubernatorial reprieve and later a royal pardon. The Prendergast family left for Chautauqua County shortly thereafter as it was outgrowing the house.[1]

Kane, an Irish immigrant, bought the house later that year. When the Revolution started, he was initially on the Patriot side, as he had been elected to the Provisional Congress of New York in 1775. After the first year of the war, he switched sides and became a Loyalist, convinced the cause was hopeless. Accordingly, his house and property were confiscated by the New York State Legislature, and the following year, in September 1778, George Washington moved in when the Continental Army wintered in the area, where they could move on either New England or New York City at short notice. Kane retreated to the safety of British lines for the remainder of the war, while his family went to Nova Scotia. He received a lifelong pension from the British in 1783, when the war ended, and returned to the Pawling area. Since he could not legally live in or repurchase his home, he lived the remainder of his life with his children.

A later owner in the early 19th century demolished all but the original kitchen wing and built the current Federal structure. It went through a variety of later uses, as an inn and a rental property owned by the local bank, but returned to single-family dwelling status late in the century. In 1946 one owner added electricity; it is not known when modern heating and plumbing were installed.[1] In the late 20th century, after it had been listed on the National Register, it was acquired by the Historical Society of Quaker Hill and Pawling and converted to its present use.