An analysis is given of the local phase space of gravity coupled to matter to second order in perturbation theory. Working in local regions with boundaries at finite distance, we identify matter, Coulomb, and additional boundary modes. The boundary modes take the role of reference frames for both diffeomorphisms and internal Lorentz rotations. Passing to the quantum level, we identify the constraints that link the bulk and boundary modes. The constraints take the form of a multi-fingered Schr"odinger equation