In the event that an employer attempts to bring directly engaged contractors onto a site in order to complete specialist work, is the contractor legally able to refuse access?

Usually the contractor has exclusive possession of the site, as stipulated in standard contracts such as JCT 2005. Thus, unless otherwise explicitly state in the contract, the actions described in the question would be considered an act of gross interference and completely unacceptable.