ABSTRACT
CeTe3 is a unique platform to investigate the itinerant magnetism in a van der Waals (vdW) coupled metal. Despite chemical pressure being a promising route to boost quantum fluctuation in this system, a systematic study on the chemical pressure effect on Ce3+(4f1) states is absent. Here, we report on the successful growth of a series of Se doped single crystals of CeTe3. We found a fluctuation driven exotic magnetic rotation from the usual easy-axis ordering to an unusual hard-axis ordering. Unlike in localized magnetic systems, near-critical magnetism can increase itinerancy hand-in-hand with enhancing fluctuation of magnetism. Thus, seemingly unstable hard-axis ordering emerges through kinetic energy gain, with the self-consistent observation of enhanced magnetic fluctuation (disorder). As far as we recognize, this order-by-disorder process in fermionic system is observed for the first time within vdW materials. Our finding opens a unique experimental platform for direct visualization of the rich quasiparticle Fermi surface deformation associated with the Fermionic order-by-disorder process. Also, the search for emergent exotic phases by further tuning of quantum fluctuation is suggested as a promising future challenge.