Significant progress has been made in anomaly detection and safety monitoring to improve the safety and security of cyber-physical systems (CPS). However, less attention has been paid to hazard recovery and mitigation. This paper proposes a combined knowledge and data driven approach, KnowSafe, for the design of safety engines that can predict and mitigate safety hazards resulting from attacks or critical faults targeting a CPS controller. We integrate the domain-specific knowledge on safety constraints and context-specific mitigation actions with machine learning (ML) techniques to estimate system trajectories in the far and near future, infer potential hazards, and generate optimal corrective actions to keep the system safe. Experimental evaluation on two realistic closed-loop testbeds for medical CPS and one actual clinical dataset in comparison to state-of-the-art attack recovery methods demonstrates the high accuracy of KnowSafe in predicting system state trajectories and potential hazards with a low false positive rate and no false negatives. Furthermore, our proposed mitigation methodology maintains the safe operation of the simulated CPS despite attacks without introducing any new hazards, with a mitigation success rate of 92.8%, which is at least 76% higher than solely rule-based (50.9%) and solely data-driven (52.7%) methods.