The correlation analysis of the independence between the behavioral influencing factors has been performed in order to find the non-independent factors. Among the Human Factors Analysis and Classification System (HFACS), the impact of the high-level factors on the bottom level is determined. This study uses empirical method, namely Structural Equation Modeling (SEM) to build the relationship model. After finishing the framework of construction safety influencing factors, we design the questionnaire survey, ultimately draw SEM model by AMOS software. The results by AMOS show that the model can effectively describe the relationship between the influence factors of which affecting construction safety and provide strong evidence for improving the engineering and construction management.