Towns are the unfortified downgrade of the Castle in the Heroes of Might and Magic series. In Heroes I and II, the player must build a fort before building any other buildings. In Heroes III, the player can build all non-dwelling buildings (with a few exeptions). In Heroes IV, the same applies, however the list is smaller due to the fact that most extra buildings are dwelling modifiers. Heroes V is the only game where a Fort is not required in the town.
In every Heroes game (except V) You can clearly see what towns have a fort build and which do not, for they change in appearance.
In Heroes V a fort only gives walls to protect ground units from reaching the inside of your city, assuming the opponents catapult hasn't destroyed your fort walls.