They HAD to go! Two Eunonymous shrubs I've had for 20 years... I liked them, but what was "supposed" to be "5 feet high and 3 feet wide" was 8 feet high and 6 feet wide. And started to get invasively sending up shoots all around the flowerbed.
Everything has a place, even if not where originally placed.