Just a thought. If you go to Edit>Preferences and the Performance Tab, what value do you have for the Raw Cache setting. I use 80GB. When you Zoom in (unless you opted for 100% previews) it has to render a full sized preview. However, once you do this the preview is temporarily stored in the cache and I set a limit of 30 days. So if I come back to the image within 30 days the full sized preview will then be instantly available. Once I start editing an image, it is very rare that I will return after 30 days. I am not saying that this is the issue but it may be worth checking some of your settings givens that others do not seem to have the issue.
Dave