Oliver and Isabel Stine purchased the property in 1915 to establish a mountain side retreat. Mrs. Stine spent a year visiting gardens in Japan before returning home to begin developing this garden. She hired two well known Japanese landscape designers to make the plans for the property.

The garden changed owners in 1932 and eventually was inherited by Mrs. Walter Gregory. She died in 1959 and the gardens were left unattended for a few years. A group of people bought the property in 1961 to use as a private retreat. They sold it to the City of Saratoga in 1966 with major restoration and new construction through 1991.

There are currently several areas including a Hill and Pond Style Garden, Tea Garden, Zen Garden and a bamboo garden.