This paper presents a video authentication system over internet protocol that is insusceptible to illumination and expression variations. The illumination and expression invariant features are extracted using multi-band feature extraction. These features are classified by a radial basis function neural network. A new adaptive decision fusion method is proposed to combine the scores from different modalities and the different frames during the authentication process. Three levels of decision fusion are carried out in the proposed adaptive decision fusion. Depending on the level of decision fusion, the level of illumination influence is taken into account during the decision making. Copyright © 2011 Inderscience Enterprises Ltd.