The assumption that the setRemoved method will never be called more than once is flawed, considering even vanilla code seems to indicate it might happen. Especially with plugins, throwing an exception is not something reasonably maintainable across all the places it *could* happen.
If it is called a second time after already having been removed due to changing dimensions, that's definitely bad, so no extra check for that
Fixes#9420
