Cheval began the building in April 1879. He claimed that he had tripped on a stone and was inspired by its shape. He returned to the same spot the next day and started collecting stones.

For the next 33 years, during his daily mail route, Cheval carried stones from his delivery rounds and at home used them to build his Palais idéal, the Ideal Palace. First he carried the stones in his pockets, then a basket and eventually a wheelbarrow. He often worked at night, by the light of an oil lamp.