Thats correct. Removing property mainly refers to theft of received but not delivered mail, property exclusively owned by USPS, or any other property inside the building or attached to the building. There simply has to be public access to the main postal lobby, thus and unless they can prove damage to the (ground) property then any reports to be investigated would be by the local authority.
Here is a small piece of quirky info too. Once mail has been delivered to an approved mail box it is still property of USPS, if you add anything into the box it is tampering with USPS mail. How many of us have slipped something into a mailbox at the home owners request so they could get access to it later? Keys, birthday party invites, maybe a catalog for your neighbor who asked for it etc etc. Technically the carrier can seize that property and return it to the PO for investigation or destruction. Doesnt happen often but it is a violation of Fed Code. One reason why we have a locked box that only we have a key.