No. The risk of lung disease is linked to people who regularly breathe construction dust over a period of time, not on the odd occasion. However, breathing construction dust even over a short period of time is not nice (for example when passing some street works) and could cause a reaction in someone who already has asthma or another existing breathing problem. It therefore should be avoided by using the right controls and making sure that members of the public are not close to the work when it is going on (see also Protecting the public: Your next move).
The Health & Safety Executive does not deal with dust from construction sites where the main concern is that it is a nuisance. The Local Authority Environmental Health Department for the area may be able to assist with this problem.