The only things that can happen are they move out of sight or you addapt. Both are very very very rare.