I replaced the starter on a 2002 XJ8. no crank after replacement, starter works (tested). How do I reset the security system so the starter will engage?

 

You'll need a trip to the dealer, not what you want to hear but it's not something an owner can do unless they have the dealer level diagnostics hard/software.
How have you come to the conclusion it's in security mode?
Have you checked the ignition relay?