I'm more frightened of being put to sleep, not so much waking up in the middle of things (nearly impossible), and I'm just about 100% sure I'll wake up at the end of things--although when you consider major surgery like this, it's something you think about in passing.
I think it's the "losing control" part of it. I've never been given anything stronger than a local anaesthetic, really.