I guess it could damage it, but it seems more likely that it would bend a pushrod. You can pull out the lifter and look at the cam with a flashlight. It is probably fine. You have rail rockers right? Mine are loose too unless they are open. I was told that was normal.