In the eighteenth century, Raynham Hall stood on a pie-shaped six-acre plot of land that stretched to Oyster Bay Harbor, located behind the house. On the opposite side of West Main Street was an apple orchard belonging to the Townsends. During the nineteenth century, the family sold off much of their property to relatives and neighbors, but retained enough land for an elegant garden on the west side of the house.

Today, the grounds of Raynham Hall Museum encompass two thirds of an acre, forming an oasis of beauty in the heart of the historic village of Oyster Bay. The gardens have been designed to correspond to the two historical periods represented by the house, the colonial and Victorian. Each is based on landscaping principles of the period and includes historically accurate plantings. The colonial garden, located at the front and west side of the c.1740 house, features brick walks bordered by daylilies and ferns, French lilacs and boxwood hedges, and ornamental elements such as teakwood benches. On the west side of the 1850’s addition is the Victorian garden. Serpentine ivy beds combine with an assortment of rhododendrons, mountain laurels and white pine trees to create a picturesque effect. Ornamental features include a cast-iron urn in the center of a formal parterre outside the dining room bay window and a pair of urns on pedestals flanking a brick terrace at the rear of the garden.