The challenge comes more from the fighting than the secrets to be honest. Secrets can net you some gear that changes how fighting can go slightly, but really only one or two items drastically change encounters. The rest are more just for enhancing specific stats/abilities It's more for lore drops