An appeal to the Planning Inspectorate to keep a fence built for “safety” at a Lostock property has been rejected.