Under NEC3 Option D: access to the Site can only be given in piecemeal due to Right of Way problems. In the assumption of the PM, idling could occur. The PM thinks that Direct and indirect labours mobilized at the moment exceeds what is the actual required for the works because construction activity is limited due to Right of Way problems.
The answer is ‘Yes, they can’, but it sounds as if they are either adding in additional constraints not in the original Works Information (a compensation event under clause 60.1 (1)) or instructing you to not to start or stop work (a CE under 60.1 (4)).
Either way, it sounds as if it is a compensation event which does not arise from your fault and hence he / she should instruct a quotation for the effect.
Sounds as if there should have been an Early Warning Notification from the Project Manager. In this absence of the Project Manager doing this If it is an on going problem you should notify an early warning to protect your interests as a 60.1(2) compensation event re access is one the compensation events the Contractor should notify. You could then instruct the Project Manager to attend a risk reduction meeting to discuss what appears to be an on-going problem and find ways to mitigate the issue.