The Horseman is in the Scarlet Monastery. The graveyard to be exact. You get the quest in there to summon him from his pumpkin shrine. He's a 70 elite so it takes a 5 man crew (of 70s) to bring him down.
I got the Magic Broom mount too, it goes a lot faster than I expected!