The main thing I can think of that would scare Neal is losing one of the few people he truly cares about, perhaps thinking that Peter, El, Mozzie, June (or Sara?) had died. In my head-canon, the prospect of getting stuck in solitary confinement for an extended period of time also scares the crap out of him.