Lol, very kind Neil, but I'm very much the amateur. I tend to think of myself as similar to the early style antiquarians, before archaeology and expert historians. I like to kick around the rare facts, the myths and pure speculation to come up with something plausible. A real expert would be very unimpressed. It requires far more rigor. I just like playing about and imagining. Since there is no archaeology to speak of, there are no right answers. I'm not even sure that 'what is a castle' is easy to pin point. I went on a castle day course at Dover and came out less sure about stuff than I went in.
Opposite the gaol and next to St Michaels were other buildings, at least one of which looks like the lower storey is stone (seen here 1776). Once the castle was no longer needed there would have been a lot of stone to hand.
You have seen it Rob. That snippet is a photo of a larger copy than yours. One worth scanning for CD as it includes details that are nowhere else but that etching. There is a painting in the Aylesford collection that captures half of the same building which is why I think that the lower part of the building is clad in stone.