Augmented reality surgery has not been successfully implemented in dental implant surgery due to the negative impact of an incorrect implant placement. This research aimed to improve the convergence between computed tomography derived teeth model and real-time stereo view of patient’s teeth to provide high registration accuracy. Enhanced iterative closest point algorithm is proposed to reduce the error caused due to matching wrong points. Weighting mechanism and median value are used to reduce alignment error caused due to matching wrong points. In addition, random sample consensus (RANSAC) algorithm is used to detect and remove the outlier. Furthermore, the current solution for dental implants did not provide the position and orientation of the surgical tool, and without this information, there is a risk of damaging adjacent structure, dental nerves, and root canals. Optical tracking device is used in the proposed solution to address this information and ensure that nerve does not get damaged during the dental implant placement surgery. While the state-of-the-art solution provided 0.44 mm registration accuracy, the proposed solution was improving it by providing 0.33 mm registration accuracy. Additionally, the proposed system can produce good results despite not having a good initialization. The processing time improved to 14 fps in comparison to the 9-fps given by state-of-the-art solution. The proposed system improved the accuracy of convergence and the processing time compared to the globally optimal-ICP algorithm. We also employed RANSAC algorithm to detect and remove the outlier on the estimation and reduce the influence of extreme points.