Here's the code in system_modules_submit() - the function called when admin/modules is saved - that skips over flushing caches if no modules are enabled or disabled: