well, it's HIS property and he can do what he wants to it.
Did the historical society ever declare the gazebo a historical monument? If not, not much they can do about it.