Since 1942, the Albany County Historical Association has been interpreting Albany's history. In 1948, the ACHA was charged with the stewardship of the historic Ten Broeck Mansion, and for more than a half-century has been maintaining, preserving and making historically accurate renovations for the benefit and enjoyment of the community, and to ensure the integrity and history that the mansion stands for. The Ten Broeck Mansion was built in 1797-98 for General Abraham Ten Broeck and his wife, Elizabeth Van Rensselaer, after a fire destroyed their previous Albany home. Ten Broeck leased five acres in what then was the Township of Watervliet from his brother-in-law, the Patroon, Stephen Van Rensselaer. It was there he built a Federal-style house with sloping lawns and formal gardens.The mansion, named "Prospect," commanded a sweeping view of the Hudson River and its daily traffic of barges and schooners along the busy trade route. The Ten Broeck family was already five generations into New World residence by the time Abraham came of age and had become prosperous and prominent in public affairs. In 1775, he served as a delegate to the Second Continental Congress. In 1777, as a brigadier general, he commanded the New York Militia at the famous Battle of Saratoga. From 1779-1783 he served as both mayor of Albany and as a member of the State Senate. Governor George Clinton appointed Ten Broeck the first judge of the Court of Common Pleas, in 1781.

Among Ten Broeck's non-governmental accomplishments were serving as the first president of the Bank of Albany, first president of the Albany Public Library, and as a trustee of Union College. Ten Broeck enjoyed Prospect for a scant 12 years before his death in 1810. His widow, Elizabeth, lived there another three years, until her death.

Over the next 30 years, the character of the house changed. It was refurbished and renovated in the then-fashionable Greek Revival style. Thomas Worth Olcott purchased the residence in 1848 and renamed it Arbour Hill, after the surrounding area which today is an Albany neighborhood known by that name.  The Olcott family, one of the most prominent in Albany banking and civic improvement circles, was responsible for the addition of the first-floor butler's pantry and the second-floor bathrooms, both of which reflect the Victorian style of the late 19th Century.

The Mansion's dominance spurred new development in Arbour Hill, with a new wealthy merchant class building homes near their businesses, utilizing the late Greek Revival, Italianate, Second Empire and other styles from the late 1840s to 1890.  In 1948, after 100 years of Olcott family ownership, the Mansion was presented by the heirs of Robert Olcott to the Albany County Historical Association.