The Castle Inn, on the south side of the Bath Road, got its name from the view it commanded of Windsor Castle. It was one of the best known inns in the Salt Hill area, although the prices were so high that a customer suggested that the landlord should change his name from Partridge to Woodcock "on account of the length of his bill".

Until the hotel closed in 1841, the old Etonians held their Montem breakfasts here every second year, alternating with the nearby Windmill Inn.

However, the Castle Inn is most famous for a tragedy that occurred there in 1773. On the 29th of March a dinner was held for the Commissioners of the Colnbrook Turnpike Trust, following which all but one of them fell ill, and fifteen died. Mrs Partridge, wife of the landlord, confessed on her deathbed that the cause had been a turtle soup, left to cool and then reheated for the dinner, in which a poisonous verdigris had formed.

The building became a private house in 1841, and a large part of it was pulled down in 1887. In the 1930s it reopened as the Salt Hill House Hotel, but it was demolished in 1964.