Baylis House was built in 1696 by Dr Gregory Hascard, Dean of Windsor. It is not known whether he designed it himself, or if it was designed by Sir Christopher Wren or John James of Greenwich.

The house has been altered and extended several times, including the addition of a third storey in 1726, and further work by John James in the 1730s.

In 1830, William and James Butt took over the tenancy and turned Baylis House into the St James Roman Catholic School, moving from Richmond because of a need for larger premises. At the time it was the only Catholic school in or around Slough. The school was largely self-sufficient, farming 99 acres and operating their own brewery, bakery and dairy.

Between 1924 and 1936 Baylis House was run as a hotel by Mr J B Whaley. It was during this time that the swimming pool was built.

Slough Borough Council bought the house and some of the surrounding land in 1939. During the war it was used as the headquarters of the ARP, and in 1941 was visited by the Duchess of Kent. After the war the grounds were developed as a sports and recreation centre.

In 1954, the house was taken over by Buckinghamshire County Council, and in 1958 was leased to the Urwick Group. Baylis House is currently in use as a conference centre.

Baylis House has twice been struck by fire. The first time was in 1954, when the repairs carried out by Buckinghamshire County Council included the removal of the third storey which had been added in 1726. There was another fire in 1973.

There is also a rumour that the house is haunted. The appearance of the ghost of a 'titled lady' in the east wing is said to foretell a disaster in the house.

In 1949, Baylis House was added to the Provisional List of Buildings of Architectural and Historical Interest. It is now a Grade I Listed Building.