I have a 2015 XJ and have an annoying inconsistent whistling sound coming from what I think is the driver’s side mirror.

Initially, I thought the door seals needed replacing, but I decided to shut my side mirrors whilst driving at 70mph and the sound went away.

Has anyone experienced this before? Is there a fix?

 

Door mirrors do make airflow between mirror and door/window. When you fold them the flow chances. I put my penny that your main door seal had moved a bit away from its groove letting air in. Door seals are not clued or anything so its very common that they move in time. Just tap them by hand. (pull one corner out to get the idea how they are mounted. (hard to explain their mounting)
Seals can go flat as well in time and lose their pressure letting air flow in. Use some door seal silicone to refresh them.

 

The presence of noise is inconsistent. I drove for two hours in the rain and not an issue. The noise came back after leaving the car for a few hours to drive it again.

Regardless of the weather, the noise makes its occasional appearance from time to time.

 

Yes, because you used the door and seal moved again a bit. -> Just a theory, but been fixed many of same issues with just seconds by tho tap with my palm.

 

Dry off the doors for a while to ensure that the seals are completely dry. Then take baby powder and rub it into the entire seal all around. Do this once every 4-6 months and your seals will stay soft.