Personally, I didn't find the Cave of Ordeals all that difficult (aside from the rooms where you can get frozen). The ball and chain pretty much wrecks any bit of difficulty most of the rooms had, there's no bosses and if you get lucky with that purple chu jelly, you can find free life refills every few levels.
That said, it's probably the most difficult thing in 3D Zelda, aside from getting that goddamn heart piece in the Goron section of Majora's Mask's moon.