Accurate fall detection for the assistance of older people is crucial to reduce incidents of deaths or injuries due to falls. Meanwhile, vision-based fall detection system has shown some significant results to detect falls. Still, numerous challenges need to be resolved. The impact of deep learning has changed the landscape of the vision-based system, such as action recognition. The deep learning technique has not been successfully implemented in vision-based fall detection system due to the requirement of a large amount of computation power and requirement of a large amount of sample training data. This research aims to propose a vision-based fall detection system that improves the accuracy of fall detection in some complex environments such as the change of light condition in the room. Also, this research aims to increase the performance of the pre-processing of video images. The proposed system consists of Enhanced Dynamic Optical Flow technique that encodes the temporal data of optical flow videos by the method of rank pooling, which thereby improves the processing time of fall detection and improves the classification accuracy in dynamic lighting condition. The experimental results showed that the classification accuracy of the fall detection improved by around 3% and the processing time by 40–50 ms. The proposed system concentrates on decreasing the processing time of fall detection and improving the classification accuracy. Meanwhile, it provides a mechanism for summarizing a video into a single image by using dynamic optical flow technique, which helps to increase the performance of image preprocessing steps.