Although the metadata, such as the header, of a piece of media carries useful information, the metadata may be tampered with for various purposes. It is therefore desirable in the context of forensic analysis that investigators are able to infer properties and information about a piece of media directly from its content without any reference to the metadata. The block size of the block operations that a piece of media has undergone can provide useful clue about the trustworthiness of the metadata and in turn reveals the integrity of the media. In this work, we proposed a novel block artifact detection method for inferring the block size of block-wise operations, such as JPEG compression, that has been applied to the media under investigation. Based on the assumption that block operation create disparities across block boundaries and those boundaries form straight lines, our method exploits the fact that intra-block variance tend to be less than inter-block variance and if most of the pixels along the same vertical line or horizontal line exhibit this relationship then the straight line is believed to be the block boundary.