Safety is the most important factor when developing software for safety-critical systems. Traditional approaches attempted to achieve safety through testing the software. However, there might be some bugs in the software not revealed in the test procedure. Formal verification is a new trend in developing safe software. In this paper, we propose a multi-phase formal approach for safety management in safety-critical software. We use timed transition Petri-net as a formal means to specify the properties of the model and their relations in each component of the software. In addition, we use the Z language to specify textual and mathematical specifications of the model, as a representative model to evaluate the proposed approach; we chose continuous infusion insulin pump (CIIP).