The thing is that one of the purpose of a CI service is to do full clean builds, to prevent false negatives and false positives caused by previous builds’ results.
It’s of course possible and whether you want to do it depends on your project, but that’s why we have the build-cache system of course 
Awesome! You might want to consider adding this as a build-cache guide, you can mostly copy paste the template for a How To post from here How to cache CocoaPods dependencies I think It’d definitely help others / give them an idea about Xamarin related build caching.
Thanks for sharing your insights!