Do you have to remove the whole door panel to switch out the master window switch or just the little plastic housing it sits in?

If its just the housing, do I pry from the back / front / panel side / or panel edge closest to the interior of the car?