Sir Grey's defence is what could probably be considered a more traditional castle. It's very standard and there is nothing special about it at all. Several of his buildings are on the outside of his castle, though some of the most important ones are inside. Most noticeably however, his Granary is on the outside of his Castle, which means you can hit his economy easily. Sir Grey doesn't tend to send constant attacks to your castle other than Spearmen and Catapults, but he does send sieges more often than other Lords.
Sir Grey can assemble a sieging force very quickly, it mainly consists of many Spearmen as well as Swordsmen and Archers. He attempts to smash down your walls with Catapults and then sends Laddermen in for any wall left. His sieges are easy to fight off by placing several towers and many Archers inside them. These help to allow the Archers to reach further than they would on wall so you can target the Catapults before they start to attack your wall. Likewise, he can build his castle quickly making him the ideal opponent for a quick 1-1 Skirmish.
His castle is very standard, and he only ever builds the same design. The instantly noticeable weak spot is the right hand wall next to his Granary, where the wall is thinner than in other places. There is very little which can be done unfortunately to completely avoid the Fire Ballistae on the two towers, however, you can use Trebuchets when creating a breach in his wall which should be out of range. You can also use Cats to protect your Trebuchets while they destroy the wall for you. Once this is done, simply charge in, straight up the Keep!
Tip: While Sir Grey does build up his castle extremely quickly, the two points which are the slowest at being built are his two Towers. If you are quick enough, you can send troops into his castle through these gaps before he places the towers.