InspectCode cache misses on fresh repo clone

Answered

Permanently deleted user

Created September 07, 2018 13:52

Hello,

I have recently implemented code inspection using the CLI Inspect Code tool (2018.2.1 on Windows) as part of our CI process. We are utilizing a branch pipeline based model using Jenkins + BitBucket Server. As part of this pipeline, each unique build creates a clone of the repository in a new directory, then proceeds with compilation, inspection, etc. During code inspection, I have observed that the first time the build is run, the cache is not utilized (i.e. the inspect is slow). If I repeat the inspection without re-cloning, the cache is utilized and the inspection is relatively quick. Ideally, since the number of changes between builds is typically small (a handful of files), I would hope that the cache is utilized for every build (fresh clone or not), however I'm not seeing this behavior.

I've tested this using the --caches-home argument and pointing it to a common location, as well as using the defaults, however in both cases I observe the same behavior.

What can I do to ensure that the cache is utilized for each unique build (in unique directories), or do I have to re-think the 'fresh-clone' aspect of my CI pipe?

Thanks

3 comments

Slava Trenogin

Created September 07, 2018 18:34

Hi Matt!

Currently, id for caches is full path of solution file.
So, if you clone your project each time to new folder, even with the same --caches-home key first run will always be cold, without caches.
You can create a feature request to provide external id for solution or use more persistent one instead of path.

Matt Murrell

Created February 14, 2020 14:05

Matt -

I struggled with this as well and it was particularly challenging for us. Our CI process runs inspections on every PR in TeamCity, and this was adding 12 minutes to our CI process. Our TeamCity build was set to always use a static directory (D:\Build\Project) for checkout, so changes to the SolutionName / path would not cause the cache misses as Slava had suggested.

After digging a little deeper I was able to confirm my suspicion that the file timestamps play a factor in caching. Because git does not keep file metadata like timestamps with the commits, [ref] every time TeamCity would clone the files from the repo it would increment the file timestamps even if the files had not changed, and this caused cache-misses in the inspectcode app.

Fortunately I was able to hack together a potential resolution. I wrote a small powershell script that iterated over every file and set the timestamps to a constant value after the build had completed, but before I ran my inspections. When this script was included in my test process, the cache was used every time and my code inspection time was reduced by 90%.

For reference, I used the NLog project to reproduce the problem and test solutions. My testing process was

Delete the local nlog folder. This would emulate TeamCity clearing the build folder.
Perform a shallow, single-branch clone from the remote repo
Perform a package restore and compile
Set Timestamps to '2020-01-01'
Inspect Code with --Caches-Home set to a known folder.

On my local machine, without running Step#4, the InspectCode process took ~8:30m consistently. However, once I put that step in place, the initial run took ~8:30m to build the cache based, and subsequent runs took ~30s.

For completeness, I'll attach the powershell script that I used to set the timestamps:

param (
  [string]$path,
  [string]$date = '2020-01-01 1:23:45.678'
)

$d = $(Get-Date $date)
$modifyfiles = Get-ChildItem -force -recurse $path | Where-Object {! $_.PSIsContainer}
foreach($object in $modifyfiles)
{
  $object.CreationTime=$d
  $object.LastAccessTime=$d
  $object.LastWritetime=$d
}

And the batch file that I used to test my implementation.

REM All my repos are in C:\Git
REM Command Line Tools are extracted to C:\Program Files\Resharper CLT\
REM My test folder is named "dummy" inside of "C:\TeamCity"

rmdir /s /q C:\TeamCity\dummy
del *.txt
git clone -b dev --single-branch "C:\Git\NLog" C:\TeamCity\dummy > git.txt
msbuild /t:Restore C:\TeamCity\dummy\src\NLOg.sln > restore.txt
msbuild /t:Build C:\TeamCity\dummy\src\NLOg.sln > build.txt
powershell -Command ".\set_timestamps.ps1" "C:\teamcity\dummy" "2020-01-01 01:23:45.678"
powershell -Command "Measure-Command { C:\Program Files\Resharper CLT\InspectCode.exe dummy\src\NLOg.sln --verbosity=verbose --caches-home=C:\teamcity\Cache --output=C:\TeamCity\report.xml > inspect1.txt }"

I'm not sure why InspectCode chooses to look at the file timestamp for change-detection, and by forcing the timestamps to a constant value, this may cause the app to miss some changes to the project file or otherwise fail to inspect correctly. I'll update the thread if I have any feedback from the script.

HTH

Other Matt

Ivan Serduk

Created August 28, 2020 17:04

ReSharper/Rider/inspectcode uses file's timestamps to identify that the file has been changed and its caches should be rebuilt. If you explicitly set timestamps for files to the same value, ReSharper will not find that they has been changed and will not rebuild caches which will cause most of ReSharper's functionality to be broken.

Starting from version 2020.1 file's content's checksums are also used to check that the file has been changed, i.e. if timestamps of cached version and new version are differrent but content's checksums are equals then ReSharper treats file as unchanged.

Please sign in to leave a comment.