Back of a 99 XK8 engine. Should be similar to yours. There are several places for chunks of metal to sit on the back of the block and scrape against the back of the flexplate.

 

Like it or not, it seems like it is time to pull the transmission and pull the flywheel for a careful examination of the bell housing and flywheel gears. Something is caught between the flywheel and engine body and/or bell housing.

 

I didn't want' to be the bearer of bad news, but since you said it Jim...I'll simply concur.

 

Need to think about that. Not sure the car is worth it.
The car is approaching 160K and it would probably be the opportunity to completely service engine and transmission.
I do not have the equipment to do this, there are no low cost experts around here to help out, the value of the car is not that high in this market. And frankly speaking, I would probably end up having it repainted as well.
Going for a biz trip a couple of days. Good opportunity to just leave it for now and think about it.

 

Instead of pulling the transmission I was able to solve the issue with...
... duct tape.

I feel like a total idiot.
Turned out the air hose between the left valve cover and the air hose broke off when placing the cover back (after the gasket change), Apperantly this causes an agonizing whistling sound.

Nevertheless, I did inspect the flywheel, bell house and the part where I found the damages really well. I couldnt find any pieces nor did I find any marks on the wheel that would indicate something grinding against it. Spinned the crank a couple of rounds and asked a friend to listen to the flywheel. Nothing.
Then turned the engine on and let it run for a bit.
Initially smoke got really bad, but I couldnt pinpoint where it was coming from. Then after about 10 minutes it went. Must have been some oil residue.
During that time we tried to pinpoint the location of the high pitch sound. It didn't sound mechanical and seemed to come from the front of the engine rather than the rear.
When inspecting the valve cover gaskets, I suddenly noticed the cracked tube. Put it back together with duct tape and the sound was gone.

In all fairness, the combination of the damage on the old starter motor and the opening it sits in, the high pitch sound and the location of the smoke did all point to a single direction.
Still I am an idiot
Thanks to you all for all your help and input. It has been usefull and maybe it will be usefull for others in the future (hopefully not).

Finally, I promissed to post some details on the tools I used for removing and installing the starter. Having used it now about 6 times, I think the method can be considered confirmed.
1. Follow JTIS for removing all connections. Note that the alternator is also connected to the main terminal. This is not specified in JTIS.
2. Remove the plastic parts (see picture, not sure what they are), located right behind the oil filter.
3. Loosen the bottom bolt of the starter (dont take it out yet). Use a couple of extensions and a 17" long socket (though I managed with a 17" normal).
4. Use the tool as displayed below (17" socket, 75mm, wobble, 2x 150mm (or 150 + 2x75), all 1/2.
You should be able to squeeze the tool between the starter and the engine/heat shield from the opening behind the oil filter. Hold the tool right below the wobble and use your other hand to guide the socket to the bolt. Get ready for some serious cursing. It will take a couple of attempts.
Getting the starter in and out is also a serious puzzle of turning and moving it around. It should come out pretty vertical from where it sits (not behind the oil filter).