For nearly 700 years, the Stourton family had owned this property before it was purchased by Henry Hoare I in 1717. Hoare demolished the original house and replaced it with a Palladian house and renamed the property Stourhead.

The 100 acre gardens were the original design of Henry Hoare II starting in 1740. The gardens are based on a pathway around the lake with contrived long views of classical temples. In 1791 Sir Richard Colt Hoare began the arboretum, mainly with deciduous trees from North America, and began the collection of rhododendrons. The present planting, however, dates mainly from the late 19th and 20th centuries. Ornamental conifers were added by the Victorians and the late Sir Henry Hoare planted many new varieties of rhododendrons.