The ground states encode the information of the topological phases of a 2-dimensional system, which makes them crucial in determining the associated topological quantum field theory (TQFT). Most numerical methods for detecting the TQFT relied on the use of minimum entanglement states (MESs), extracting the anyon mutual statistics and self statistics via overlaps and/or the entanglement spectra. The MESs are the eigenstates of the Wilson loop operators, and are labeled by the anyons corresponding to their eigenvalues. Here we revisit the definition of the Wilson loop operators and MESs. We derive the modular transformation of the ground states purely from the Wilson loop algebra, and as a result, the modular S and T matrices naturally show up in the overlap of MESs. Importantly, we show that due to the phase degree of freedom of the Wilson loop operators, the MES-anyon assignment is not unique. This ambiguity obstructs our attempt to detect the topological order, that is, there exist different TQFTs that cannot be distinguished solely by the overlap of MESs. In this paper, we provide the upper limit of the information one may obtain from the overlap of MESs without other additional structure. Finally, we show that if the phase is enriched by rotational symmetry, there may be additional TQFT information that can be extracted from overlap of MESs.