The wilderness golems seem very hard to kill, until you learn how to use the 'block' ability (hold Q). This ability appears to make you immune to frontal attacks while it is up.

The golems' attacks hit heavily, but otherwise they are easy opponents - they are slow, with big pauses between attacks, and don't move around unless you do.

So just find a solitary golem, stand in front of it, get in lots of attacks, then hold Q during its attack animation. Rinse and repeat. You will not need any potions unless you mistime the block.