Like you, I've never broken a bone or had anything but a minor surgery. I think a terrible high ankle sprain I got as a kid was probably my worst. Seemed bad at the time, but was nothing compared to most other things.
As an aside, I have a phobia of getting kidney stones. I've never gotten them and have no family history, so I'm low risk. But something about the stories from people who've had them is horrifying to me. I can make my palms sweat heavily just thinking about them.