ABSTRACT
Layered honeycomb cobaltates are predicted as promising for realizing the Kitaev quantum spin liquid, a many-body quantum entangled ground state characterized by fractional excitations. However, they exhibit antiferromagnetic ordering at low temperatures, hindering the expected quantum state. We demonstrate that controlling the trigonal distortion of CoO6 octahedra is crucial to suppress antiferromagnetic order through enhancing frustration in layered honeycomb cobaltates. Using heterostructure engineering on Cu3Co2SbO6 thin films, we adjust the trigonal distortion of CoO6 octahedra and the resulting trigonal crystal field. The original Néel temperature of 16 kelvin in bulk Cu3Co2SbO6 decreases (increases) to 7.8 kelvin (22.7 kelvin) in strained Cu3Co2SbO6 films by decreasing (increasing) the magnitude of the trigonal crystal fields. The first-principles calculation suggests the enhancement of geometrical frustration as the origin of the suppression of antiferromagnetism. This finding supports the potential of layered honeycomb cobaltate heterostructures and strain engineering in realizing extremely elusive quantum phases of matter.