Usually the contractor has exclusive possession of the site, as stipulated in standard contracts such as JCT 2005. Thus, unless otherwise explicitly state in the contract, the actions described in the question would be considered an act of gross interference and completely unacceptable.