Existing work on quantum causal structure assumes that one can perform arbitrary operations on the systems of interest. But this condition is often not met. Here, we extend the framework for quantum causal modelling to situations where a system can suffer sectorial constraints, that is, restrictions on the orthogonal subspaces of its Hilbert space that may be mapped to one another. Our framework (a) proves that a number of different intuitions about causal relations turn out to be equivalent; (b) shows that quantum causal structures in the presence of sectorial constraints can be represented with a directed graph; and (c) defines a fine-graining of the causal structure in which the individual sectors of a system bear causal relations. As an example, we apply our framework to purported photonic implementations of the quantum switch to show that while their coarse-grained causal structure is cyclic, their fine-grained causal structure is acyclic. We therefore conclude that these experiments realize indefinite causal order only in a weak sense. Notably, this is the first argument to this effect that is not rooted in the assumption that the causal relata must be localized in spacetime.