Here's the code in system_modules_submit() - the function called when admin/modules is saved - that skips over flushing caches if no modules are enabled or disabled:
Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution 3.0 Unported License.