It massively depends on what aperture you are using.
I am pretty certain there are hundreds of articles on the internet predicting all manner of doom and gloom if you go anywhere near it with anything other than a lens cloth. A clean corner of a cotton shirt used as one off will do no harm at all to even the cheapest of modern lenses, the coatings are just a bit tougher then that. As already said, no they won't affect your images either. I cant understand why youd go on holiday and not take some cleaning kit :s dust will product flare and depending on what your shooting show up on the shot. Generally dust spots are a common pain to photographers but flare isnt to east to remedy.
I tend to clean my gear the night before a planned shoot.
the 2 articles has really changed my view on dust on the lens, i used to be really OCD about it