Nothing. Nothing went wrong with James Spader.
For all his flaws - including the obsessive compulsive disorder people attribute to him being difficult to work with, nothing went wrong with him.
To answer your question though, Hollywood ‘happened’ to him.
James is a man who refused to comform to industry standard. He has been accused of being typecast (which is true for all intents and purposes), something Hollywood neither celebrates nor rewards.
He is one of those rare actors who understood, early, that the most interesting characters live slightly to the left of likeable. The smug predator in Pretty in Pink. The voyeur in sex, lies, and videotape (I could watch this everyday). The lawyer who turned a courtroom into a personal theatre in Boston Legal. He kept finding characters who were wrong in fascinating ways, and he played them without apology or softening, and Hollywood has no real category for that.
An enigma, holding all his cards close to his chest and barely giving anything away - and that kind of precision makes people uncomfortable when it doesn't come packaged with the humility of someone who needs the job.
But he had a good run and will forever be one of, if not the best talent Hollywood denied us on a full scale.