Grazing exclusion is one of the common grassland management strategies to restore degraded grasslands. The effectiveness of grazing exclusion on sequestering soil organic carbon, increasing total nitrogen and improving soil biological activity has been documented in literature. Few studies, however, have examined the responses of phosphorus (P) fractions and arbuscular mycorrhizal fungi (AMF) diversity to long-term grazing exclusion. In this study, the variations of soil chemical properties, the status of inorganic and organic P fractions in the rhizosphere soil, and the AMF diversity in roots of Leymus chinensis, Stipa krylovii and Cleistogenes squarrosa and in bulk soils were investigated in continuously grazed and ungrazed paddocks (exclusion from grazing for 10-12 years) on typical and meadow steppes in Inner Mongolia, aiming to evaluate the effectiveness of grazing exclusion in improving AMF diversity and soil P status. Grazing exclusion altered plant species compositions and increased aboveground biomass and ground cover, resulting in increased concentrations of soil organic carbon and total nitrogen. The concentration of total phosphorus increased in typical steppes but reduced in meadow steppes, while the concentrations of available P and most P fractions remained unchanged or reduced following 10-12 years of grazing exclusion. Grazing exclusion improved AMF colonization in meadow steppes, but not in typical steppes, attributing to the differences in soil quality, plant species, and AMF phylotypes between two types of steppes. AMF diversity was positively correlated with soil pH, concentrations of soil total nitrogen, total organic carbon, total P, Ca10-P, medium labile organic P, and the activity of alkaline phosphatase, indicating that, on semi-arid steppes in northern China, improved soil conditions would increase the AMF diversity, thus enhancing the productivity of the steppe ecosystem. However, changes of soil AMF phylotypes due to overgrazing would be detrimental to this fragile ecosystem.