The fort represents the military power of your town.
Here you can buy cannonballs for ship battles. The price per cannonball increases with each town level. Upgrading the fort counteracts this price increase.