Sentry rating is used to reduce the effectiveness of Spy actions on you by others. The higher your Sentry rating, the less the chance is that they'll successfully recon or sabotage you, and the higher the Spy rating they will need to see your gold.
To gain Sentry rating, you have to first train Sentries from untrained soldiers (which gives you a small addition to your Sentry rating) and then buy tools to improve their effectiveness. You can further upgrade your Sentry level by increasing your covert skill which improves both Sentry and Spy effectiveness. You can train sentries and also upgrade your sentry level by increasing your overall covert skill level in the Training tab. Sentry tools are listed in the Armoury.
The formula for Sentry rating then is
a x (b x c) x d x e
Where
a = Your number of sentries
b = Your number of sentry tools
c = The strength of those tools
(there can be multiple b's and c's, as you can have different types of tools)
d = Your covert skill level
e = 1 or your race bonus (Undead have 35% sentry bonus – because the never sleep)
If you have more tools than Sentries, the unused tools will not count in this formula.