When gravity is sourced by a quantum system, there is tension between its role as the mediator of a fundamental interaction, which is expected to acquire nonclassical features, and its role in determining the properties of spacetime, which is inherently classical. Fundamentally, this tension should result in breaking one of the fundamental principles of quantum theory or general relativity, but it is usually hard to assess which one without resorting to a specific model. Here, we answer this question in a theory-independent way using General Probabilistic Theories (GPTs). We consider the interactions of the gravitational field with a single matter system, and derive a no-go theorem showing that when gravity is classical at least one of the following assumptions needs to be violated: (i) Matter degrees of freedom are described by fully non-classical degrees of freedom; (ii) Interactions between matter degrees of freedom and the gravitational field are reversible; (iii) Matter degrees of freedom back-react on the gravitational field. We argue that this implies that theories of classical gravity and quantum matter must be fundamentally irreversible, as is the case in the recent model of Oppenheim et al. Conversely if we require that the interaction between quantum matter and the gravitational field are reversible, then the gravitational field must be non-classical.