Improving Visual Studio and MsBuild Speed for Large Solution
Posted on: April 13, 2016
I am working on a side project that is a single solution with 51 projects. The amount of project is considered "big" for 2016 while it was still considered "medium" few years ago. For some reason, Visual Studio doesn't handles very well project with more than 50 projects. I could refactor the solution by consolidating some projects and having a single project for unit testing instead of 12. Nevertheless, this take some time and before optimizing the design of the solution, let's start by understand what is happening.
First, we need some basic metric. One useful extension to add in Visual Studio is the Build Monitor Extension by Daniel Vinntreus. This will give you an additional Output with the time of each project to be compiled. The second tool is also free, it is called Process Monitor. This can be download from Microsoft TechNet website. This tool lets you see what the process write on the hard drive (and more). Here was the data from both of these tool.
100h 00m 01s 253ms -- MySqlToMsSql200h 00m 03s 898ms -- ModelContracts300h 00m 04s 277ms -- ComplexTypes400h 00m 04s 901ms -- CrossLayer500h 00m 00s 871ms -- TestHelpers600h 00m 01s 139ms -- ComplexTypesUnitTest700h 00m 01s 248ms -- ValueObjects800h 00m 03s 053ms -- CrossLayerUnitTest900h 00m 01s 481ms -- ComplexTypesWithValueObjects1000h 00m 03s 586ms -- ComplexTypesWithValueObjectsUnitTest1100h 00m 05s 555ms -- ValueObjectsUnitTest1200h 00m 05s 747ms -- Model1300h 00m 01s 884ms -- ModelBuilders1400h 00m 10s 744ms -- ModelUnitTest1500h 00m 12s 927ms -- ViewModel1600h 00m 12s 527ms -- ModelBuildersTest1700h 00m 03s 407ms -- Mapping1800h 00m 14s 751ms -- DataAccess1900h 00m 03s 980ms -- ViewModelBuilders2000h 00m 10s 642ms -- MappingUnitTest2100h 00m 10s 708ms -- ViewModelUnitTest2200h 00m 11s 235ms -- Services2300h 00m 12s 411ms -- WebServicesCore2400h 00m 25s 328ms -- DataAccessMigration2500h 00m 25s 943ms -- DataAccessUnitTest2600h 00m 03s 291ms -- ScriptCleanExpiredOrders2700h 00m 16s 434ms -- ServicesIntegrationTest2800h 00m 18s 438ms -- ServicesUnitTest2900h 00m 20s 570ms -- IoC3000h 00m 08s 155ms -- ScriptDailyPerformance3100h 00m 05s 538ms -- ScriptBuyOrders3200h 00m 08s 966ms -- ScriptInitializeCache3300h 00m 08s 522ms -- ScriptEndContests3400h 00m 09s 063ms -- ScriptIndiceMarketPoint3500h 00m 15s 038ms -- ScriptBotRegisterContest3600h 00m 15s 665ms -- ScriptSymbolChange3700h 00m 12s 100ms -- ScriptFindUpcomingSplit3800h 00m 12s 101ms -- ScriptFindUpcomingSymbolRename3900h 00m 12s 011ms -- ScriptBotBuyContest4000h 00m 12s 070ms -- ScriptBuyShortOrders4100h 00m 14s 895ms -- ScriptEndOfDayBadges4200h 00m 21s 934ms -- FrontEndSharingLayer4300h 00m 12s 243ms -- ScriptUpdateCompanyInformation4400h 00m 23s 498ms -- ScriptSellShortOrders4500h 00m 23s 583ms -- ScriptSellOrders4600h 00m 34s 100ms -- ScriptSymbolSplit4700h 00m 34s 153ms -- ScriptInterdayStatisticPortefolios4800h 00m 30s 416ms -- DataAccessIntegrationTest4900h 00m 29s 488ms -- WebSite5000h 00m 08s 063ms -- Architectures5100h 00m 08s 267ms -- WebSiteUnitTest -- Time Elapsed: 00h 02m 51s 635ms
To get these statistics, I first clean up the solution to have the build rebuild everything. The total amount of time is 2 minutes 51 seconds. A lot of time is one project starting with "Script" which are webjob that run in the background. These ones are in Visual Studio under a folder and could be disabled in the future. This is something I was not doing and thus wasting a lot of build time when working on the main project : the website. Process Monitor is also educational by showing how many bytes is written when building the solution. To do so, open Process Monitor, click Filter (Ctrl+L) and add Visual Studio process (Devenv.exe) and MsBuild (msbuild.exe).
This give us 893 megs written. I am on a SSD drive so it still not bad, but to be honest quite a lot of writing. From here, I noticed few things. First, I have a lot of bin folder with the same files. Second, that we are rebuilding the same file because they referenced them. To improve, I decided to edit all projects to output on the same bin folder. Third, the jobs folder that contains all scripts are heavy on the writing.
I decided to have every project to output in the bin folder of the website. The reason is that I have IIS taking the bin folder as source so every time I build I can just refresh the browser to get the website with the latest version without deploying. After that, I went into all projects references and click reference to other projects and changing the Copy Local to false.
From there, I cleaned up everything (all bin folder empty) and re-build everything to see how the performance improved. First, the Build Monitor extension is showing some improvements:
100h 00m 00s 505ms -- ComplexTypes200h 00m 00s 601ms -- ModelContracts300h 00m 00s 352ms -- ComplexTypesUnitTest400h 00m 01s 453ms -- MySqlToMsSql500h 00m 01s 383ms -- ValueObjects600h 00m 03s 002ms -- CrossLayer700h 00m 01s 402ms -- ComplexTypesWithValueObjects800h 00m 01s 488ms -- CrossLayerUnitTest900h 00m 01s 940ms -- TestHelpers1000h 00m 02s 924ms -- Model1100h 00m 01s 637ms -- ValueObjectsUnitTest1200h 00m 01s 806ms -- ComplexTypesWithValueObjectsUnitTest1300h 00m 01s 184ms -- ModelBuilders1400h 00m 03s 282ms -- ViewModel1500h 00m 05s 198ms -- ModelUnitTest1600h 00m 02s 993ms -- Mapping1700h 00m 05s 746ms -- ModelBuildersTest1800h 00m 03s 516ms -- ViewModelBuilders1900h 00m 06s 564ms -- DataAccess2000h 00m 00s 788ms -- ViewModelUnitTest2100h 00m 03s 636ms -- MappingUnitTest2200h 00m 03s 634ms -- Services2300h 00m 04s 583ms -- DataAccessMigration2400h 00m 04s 983ms -- DataAccessUnitTest2500h 00m 04s 610ms -- ServicesIntegrationTest2600h 00m 06s 222ms -- IoC2700h 00m 06s 657ms -- WebServicesCore2800h 00m 11s 568ms -- ServicesUnitTest2900h 00m 04s 573ms -- ScriptIndiceMarketPoint3000h 00m 06s 860ms -- DataAccessIntegrationTest3100h 00m 04s 658ms -- ScriptCleanExpiredOrders3200h 00m 04s 715ms -- ScriptEndContests3300h 00m 05s 408ms -- ScriptInitializeCache3400h 00m 05s 637ms -- ScriptDailyPerformance3500h 00m 01s 181ms -- ScriptBuyOrders3600h 00m 01s 760ms -- ScriptSymbolChange3700h 00m 01s 831ms -- ScriptBotRegisterContest3800h 00m 01s 770ms -- ScriptFindUpcomingSplit3900h 00m 01s 766ms -- ScriptFindUpcomingSymbolRename4000h 00m 07s 225ms -- FrontEndSharingLayer4100h 00m 09s 518ms -- ScriptEndOfDayBadges4200h 00m 09s 531ms -- ScriptBuyShortOrders4300h 00m 09s 540ms -- ScriptBotBuyContest4400h 00m 08s 867ms -- ScriptUpdateCompanyInformation4500h 00m 11s 291ms -- ScriptSymbolSplit4600h 00m 11s 304ms -- ScriptSellShortOrders4700h 00m 10s 675ms -- ScriptSellOrders4800h 00m 02s 717ms -- ScriptInterdayStatisticPortefolios4900h 00m 10s 860ms -- WebSite5000h 00m 03s 206ms -- WebSiteUnitTest5100h 00m 03s 952ms -- Architectures -- Time Elapsed: 00h 01m 32s 275ms
The time to build is cut by half. This is already better. If we take the Process Monitor we can see the reason: we write only 51 megs.
Finally, if I unload all jobs project (scripts) one, I have a build time of 1m13sec. Not a huge improvement, but still 20 seconds less! The initial performance of 2 minutes 51 seconds to 1 minute 13 secconds is quite appreciable. With all these changes some problems raised. First, when pushing the code in the continuous integration (ci) environment, the build server is not able to build the whole solution. This is because the build server builds the startup project which doesn't copy local any references. The second problem is when you deploy. Visual Studio Publish mechanism build the main project too, whence the same consequences. So, we need to add additional steps to build everything and we will come back to have some performance lost.
An other direction is to remove the most possible projects. This approach is fine but limited to what you can group together. For example, I have 1 web project, and about 14 web jobs. This mean a minimum of 15 projects. If we want to divide unit tests from the code, we can add 1 more project. If we want to share webjobs and website logic we add one more project. Still, we have half the number of project and while working on the shared tier and website, it is always possible to unload from the main solution every webjob projects. The best way to move everything is to create a shared project, that I called "ApplicationTier". The website project remains the same but refers this new project. Inside Visual Studio, we need to go one by one in each project to drag-and-drop all files in a folder with the same name of the project. The final result is easy to read and consolidate a lot of project within one but with familiar structure. At the end, the result was very impressive. Instead of taking 2 minutes 51 seconds the build time was at 54 seconds.
100h 00m 00s 775ms -- MySqlToMsSql200h 00m 06s 799ms -- ApplicationTier300h 00m 06s 064ms -- DataAccessMigration400h 00m 03s 400ms -- ApplicationTierUnitTest500h 00m 09s 714ms -- WebServicesCore600h 00m 11s 656ms -- WebSite700h 00m 02s 612ms -- ScriptCleanExpiredOrders800h 00m 02s 312ms -- ScriptFindUpcomingSymbolRename900h 00m 05s 217ms -- ScriptEndContests1000h 00m 05s 271ms -- ScriptBuyShortOrders1100h 00m 05s 345ms -- ScriptBuyOrders1200h 00m 03s 508ms -- ScriptFindUpcomingSplit1300h 00m 07s 308ms -- ScriptDailyPerformance1400h 00m 08s 350ms -- ScriptSymbolChange1500h 00m 08s 447ms -- ScriptBotRegisterContest1600h 00m 03s 731ms -- ScriptEndOfDayBadges1700h 00m 03s 327ms -- ScriptSymbolSplit1800h 00m 04s 775ms -- ScriptBotBuyContest1900h 00m 04s 728ms -- ScriptUpdateCompanyInformation2000h 00m 02s 835ms -- ScriptSellOrders2100h 00m 02s 907ms -- ScriptSellShortOrders2200h 00m 01s 714ms -- ScriptInterdayStatisticPortefolios2300h 00m 01s 608ms -- ScriptInitializeCache2400h 00m 01s 579ms -- ScriptIndiceMarketPoint2500h 00m 04s 239ms -- Architectures2600h 00m 05s 405ms -- WebSiteUnitTest --  Time Elapsed: 00h 00m 54s 059ms
By reducing the amount of project, we have a lot less references that needs to be cut. The number of megs written on disk is about 550 megs now. The main bottle neck is all the scripts for webjobs. Since all jobs are just entry points to the ApplicationTier, if they share the same bin folder, it will reduce by a lot the building time. This is because that the first script project to build will get the binary files in the bin folder, and subsequent scripts will just build the executable project without building again the references. The result is the following 31 seconds, mainly because only 196 megs go written on the disk.
100h 00m 01s 633ms -- MySqlToMsSql200h 00m 06s 483ms -- ApplicationTier300h 00m 06s 696ms -- DataAccessMigration400h 00m 01s 458ms -- ApplicationTierUnitTest500h 00m 08s 610ms -- WebServicesCore600h 00m 09s 298ms -- WebSite700h 00m 01s 044ms -- ScriptCleanExpiredOrders800h 00m 00s 719ms -- ScriptFindUpcomingSymbolRename900h 00m 01s 140ms -- ScriptFindUpcomingSplit1000h 00m 02s 132ms -- ScriptDailyPerformance1100h 00m 02s 282ms -- ScriptBuyOrders1200h 00m 02s 216ms -- ScriptBotRegisterContest1300h 00m 02s 477ms -- ScriptBuyShortOrders1400h 00m 02s 295ms -- ScriptSymbolChange1500h 00m 00s 771ms -- ScriptEndOfDayBadges1600h 00m 02s 619ms -- ScriptEndContests1700h 00m 00s 811ms -- ScriptUpdateCompanyInformation1800h 00m 00s 858ms -- ScriptBotBuyContest1900h 00m 00s 764ms -- ScriptSymbolSplit2000h 00m 03s 963ms -- ScriptSellShortOrders2100h 00m 04s 022ms -- ScriptSellOrders2200h 00m 03s 957ms -- ScriptInitializeCache2300h 00m 04s 030ms -- ScriptInterdayStatisticPortefolios2400h 00m 03s 683ms -- ScriptIndiceMarketPoint2500h 00m 04s 649ms -- WebSiteUnitTest2600h 00m 01s 533ms -- Architectures -- Time Elapsed: 00h 00m 31s 471ms
I quickly started the RamDisk tool to see if having the scripts bin folder could benefit of it and I haven't see any improvement. Finally, I am pretty happy of the end result. I can always unload all scripts and this can be done easily since they are inside a folder. Inside Visual Studio, right click the folder that contain these folders and click "unload project". This will unload all of them in 1 operation. For further optimization, we could also unload the migration project and unit tests and by doing so having less than 20 seconds of total build time.