When making changes to a circuit, whether you add or remove components, or just re-wiring things, it is best to remove power first. 

There are a few good reasons for that. While you are doing the changes, you might:

  1. Miswrite, introducing a short circuit.
  2. Mistakenly plug external voltage to the wrong pin on the Arduino
  3. Place the circuit to an undesirable state (like having a button in "ON" position when you think its "OFF")
  4. Forget basic things like inserting a resistor in parallel with an LED.

All those things can potentially damage the Arduino and can be prevented by 

  1. removing power, 
  2. making the changes, 
  3. inspecting and verifying the changes, 
  4. and then applying power

However, it is not uncommon people making changes to the circuit while the Arduino is powered up and operating in order to speed up their work. An example of that is how I connected external power to feed the servo motors while the circuit was in operation, in this lecture.  This is ok if a few conditions are met:

  1. The circuit is small
  2. You can verify as you make the changes that your changes are correct and very unlikely to introduce any of the problems I mentioned above
  3. In the case of external power, you have high degree of confidence that the power source is compatible with the circuit.

In the case of the servo circuit in the lecture, all these conditions are met, so I did not feel that I was going to damage anything. 

I also wanted to make it easy to compare the before and after operation of the servos. In the lecture, it is easy to notice how the sound coming from the servos became a little louder as soon as I applied the external power, helping us to understand the effect of the additional power I provided. It would be harder to notice the difference if I had removed power from the Arduino, connected external power to the breadboard, and then re-powered to the Arduino because of the amount of time it would have taken to make this change.