Around the end of the 18th century the Rev John Cleland (Rector of Newtownards, County Down) became tutor to the young Lord Castlereagh and subsequently acted as Agent for the Londonderry Estates. Cleland originally lived in Newtownards but acquired land at Stormont on his marriage to Esther Jackson who came from the area.

Cleland subsequently added to his holding and around 1830 built a house at Stormont. This was in traditional mid or late Georgian style with a small fringed meadow at the front and an orchard on the hillside to the north west.

In 1858 the Cleland family commissioned the local architect Thomas Turner to convert the existing plain dwelling into a flamboyant castle. To what extent the original house survives is not clear. Some historical evidence suggests that the symmetrical five-bay block facing south contains the shell of the Georgian dwelling, the new structure having been given a "baronial" character with turrets, battlements, bartizans with conical caps, iron cresting and weather vanes. The Cleland Monogram was used on the shields held by the snarling stone gryphons which still guard the main entrance to the Castle.

The 1850's also saw extensive development of the grounds which now extended down to the main Upper Newtownards Road. The castle's apartments included a ballroom 64 feet by 24 feet, a drawing room 36 feet by 18 feet, and other fine reception rooms, with 14 family bedrooms, besides dressing rooms and bathrooms, servants' quarters and offices.

At some stage Stormont Castle was rented by Mr Charles E Allen JP a director of the shipbuilding firm of Workman & Clark Limited (then second only to Harland & Wolff). On his moving away from Belfast the Castle became vacant, and in April 1921 both it and the surrounding land was offered at auction, but withdrawn when no bid higher than £15,000 was obtained.

Later, in 1921, it was acquired with 235 acres of land as a site for the Parliament Buildings of the new Northern Ireland state. On September 20th that Parliament resolved that "Stormont Castle demesne shall be the place where the new Parliament House and Ministerial Buildings shall be erected, and as the place to be determined as the seat of the Government of Northern Ireland as and when suitable provision has been made therefore."

While there was initial uncertainty about the use to be made of Stormont Castle itself it was later decided that it should become the official residence of the Prime Minister. Sir James Craig (later Lord Craigavon) lived there until 1940, when he moved out to make more room for officials engaged in War work. Lord Craigavon was succeeded in office by Mr J Andrews and thereafter by Lord Brookeborough. While both had offices in the Castle no Prime Minister resided there with any regularity between 1940 to 1969.

On the arrival in office of Captain Terence O'Neill in 1963, substantial reinstatement and improvement works were carried out. These included the removal of an ugly glass entrance canopy and the restoration of the old ballroom as an improved Cabinet Room. In those days the Prime Minister occupied what is still the Secretary of State's office, with the Secretary of the Cabinet using the other major front room on the ground floor.

Captain O'Neill as Prime Minister resided, when in Belfast, at nearby Stormont House, originally built as a residence for the Speakers of the Northern Ireland House of Commons. His successor, Major James Chichester-Clarke had premises on the first floor converted into a self-contained flat and regularly stayed there. What is now the office of the Permanent Under Secretary of the Northern Ireland Office (NIO) remained in use for a substantially longer time as a Secretary of State's dining room. Since 1974, when Northern Ireland reverted to direct rule from Westminister, the Castle has been the administrative Headquarters for successive Secretaries of State.