I should of known that those towers are in the place where I was getting bandit's treasure knowledge. There is only two of them, so , it took a while till I've got the knowledge.

took me ages to find it, it looks exactly like a tree and has a barbed fence going around the base, couldn't tell it apart

Same place as Bandit Treasure Chest and Bandit Treasure Wagon, you can get knowledge 3 of them same time. It takes 2:30 min to respawn all of them.