Some of their members could be former servants. If they know the layout, that makes a raid far more efficient.

[ Says a man who's led his share of attacks, though they were decidedly the more destructive variety. Avia was more of a blunt instrument than a precision one. ]

While some things change, buildings would be hard to move. Scouts could've been used to get a sense of guard rotation, force of arms, gaps in security...

[ Someone has definitely done this dance before. ]