You could probably do something along the following lines:
- run: find ~/Library/Developer/Xcode/DerivedData -exec touch -t 2004210000 {} \; || true
If you place this step after your cache restore, then it will set all directories and files within DerivedData to the date specified (in YYMMDDHHMM format).
The || true at the end of the command will prevent the job from failing if DerivedData is empty (for example if there is no cache to restore).
I tested this myself and saw marginal gains, though the app I was testing is extremely small, so I would be interested to see if this benefits users with larger apps.