The small castle of Pafos rises in the Western part of the port of the city. Its characteristics today are the result of various interventions. The Frankish rulers built the castle in the mid-13th century to replace the Byzantine fort of "Saranta Kolones" which was destroyed after the earthquake of 1222 AD. It seems that the Frankish rulers originally erected two towers: the castle at the port and a circular tower, traces of which can be seen 50 meters West of the existing tower. A dividing wall connected these two towers. When the Genoese conquered the towers in 1373 AD, they made some modifications and also modelled its moat. According to some medieval historians, both towers were in use in the 15th century. Towards the end of the century, however, a strong earthquake destroyed one of the two towers. Just before the Ottoman conquest of 1570, the Venetians, who controlled Cyprus at the time, destroyed what was left of the two towers with the use of explosives, so that the towers were not used by the Ottomans. What survives today is the 1780 Ottoman restoration of the Western tower of the Franks with its Venetian additions.

The ground floor consists of a central hall with small rooms on each of its two long sides, which were used as prison cells during the Ottoman occupation period. Two small underground cells were used for long-term prisoner imprisonment. The small Turkish garrison lived in the rooms on the upper floor and used the central room as a mosque. On the roof of the castle there are twelve battlements, which received a corresponding number of cannons. The Ottomans removed the cannons in 1878, when they handed over the administration of the island to the British. From the beginning of British rule the castle was used as a salt warehouse until 1935 when it was declared a historical monument.