Improving Visual Studio and MsBuild Speed for Large Solution

I am working on a side project that is a single solution with 51 projects. The amount of project is considered “big” for 2016 while it was still considered “medium” few years ago. For some reason, Visual Studio doesn’t handles very well project with more than 50 projects. I could refactor the solution by consolidating some projects and having a single project for unit testing instead of 12. Nevertheless, this take some time and before optimizing the design of the solution, let’s start by understand what is happening.

First, we need some basic metric. One useful extension to add in Visual Studio is the Build Monitor Extension by Daniel Vinntreus. This will give you an additional Output with the time of each project to be compiled. The second tool is also free, it is called Process Monitor. This can be download from Microsoft TechNet website. This tool lets you see what the process write on the hard drive (and more). Here was the data from both of these tool.

- 00h 00m 01s 253ms	-- MySqlToMsSql --
 - 00h 00m 03s 898ms	-- ModelContracts --
 - 00h 00m 04s 277ms	-- ComplexTypes --
 - 00h 00m 04s 901ms	-- CrossLayer --
 - 00h 00m 00s 871ms	-- TestHelpers --
 - 00h 00m 01s 139ms	-- ComplexTypesUnitTest --
 - 00h 00m 01s 248ms	-- ValueObjects --
 - 00h 00m 03s 053ms	-- CrossLayerUnitTest --
 - 00h 00m 01s 481ms	-- ComplexTypesWithValueObjects --
 - 00h 00m 03s 586ms	-- ComplexTypesWithValueObjectsUnitTest --
 - 00h 00m 05s 555ms	-- ValueObjectsUnitTest --
 - 00h 00m 05s 747ms	-- Model --
 - 00h 00m 01s 884ms	-- ModelBuilders --
 - 00h 00m 10s 744ms	-- ModelUnitTest --
 - 00h 00m 12s 927ms	-- ViewModel --
 - 00h 00m 12s 527ms	-- ModelBuildersTest --
 - 00h 00m 03s 407ms	-- Mapping --
 - 00h 00m 14s 751ms	-- DataAccess --
 - 00h 00m 03s 980ms	-- ViewModelBuilders --
 - 00h 00m 10s 642ms	-- MappingUnitTest --
 - 00h 00m 10s 708ms	-- ViewModelUnitTest --
 - 00h 00m 11s 235ms	-- Services --
 - 00h 00m 12s 411ms	-- WebServicesCore --
 - 00h 00m 25s 328ms	-- DataAccessMigration --
 - 00h 00m 25s 943ms	-- DataAccessUnitTest --
 - 00h 00m 03s 291ms	-- ScriptCleanExpiredOrders --
 - 00h 00m 16s 434ms	-- ServicesIntegrationTest --
 - 00h 00m 18s 438ms	-- ServicesUnitTest --
 - 00h 00m 20s 570ms	-- IoC --
 - 00h 00m 08s 155ms	-- ScriptDailyPerformance --
 - 00h 00m 05s 538ms	-- ScriptBuyOrders --
 - 00h 00m 08s 966ms	-- ScriptInitializeCache --
 - 00h 00m 08s 522ms	-- ScriptEndContests --
 - 00h 00m 09s 063ms	-- ScriptIndiceMarketPoint --
 - 00h 00m 15s 038ms	-- ScriptBotRegisterContest --
 - 00h 00m 15s 665ms	-- ScriptSymbolChange --
 - 00h 00m 12s 100ms	-- ScriptFindUpcomingSplit --
 - 00h 00m 12s 101ms	-- ScriptFindUpcomingSymbolRename --
 - 00h 00m 12s 011ms	-- ScriptBotBuyContest --
 - 00h 00m 12s 070ms	-- ScriptBuyShortOrders --
 - 00h 00m 14s 895ms	-- ScriptEndOfDayBadges --
 - 00h 00m 21s 934ms	-- FrontEndSharingLayer --
 - 00h 00m 12s 243ms	-- ScriptUpdateCompanyInformation --
 - 00h 00m 23s 498ms	-- ScriptSellShortOrders --
 - 00h 00m 23s 583ms	-- ScriptSellOrders --
 - 00h 00m 34s 100ms	-- ScriptSymbolSplit --
 - 00h 00m 34s 153ms	-- ScriptInterdayStatisticPortefolios --
 - 00h 00m 30s 416ms	-- DataAccessIntegrationTest --
 - 00h 00m 29s 488ms	-- WebSite --
 - 00h 00m 08s 063ms	-- Architectures --
 - 00h 00m 08s 267ms	-- WebSiteUnitTest --
Time Elapsed: 00h 02m 51s 635ms 

To get these statistics, I first clean up the solution to have the build rebuild everything. The total amount of time is 2 minutes 51 seconds. A lot of time is one project starting with “Script” which are webjob that run in the background. These ones are in Visual Studio under a folder and could be disabled in the future. This is something I was not doing and thus wasting a lot of build time when working on the main project : the website. Process Monitor is also educational by showing how many bytes is written when building the solution. To do so, open Process Monitor, click Filter (Ctrl+L) and add Visual Studio process (Devenv.exe) and MsBuild (msbuild.exe).
One the filter is done, be sure to clear everything (Ctrl+X) and start building. One the build is done. Go back in Process Monitor and go int Tools > File Summary and get everything sorted by Folder. You can dive in and see what happened.


This give us 893 megs written. I am on a SSD drive so it still not bad, but to be honest quite a lot of writing. From here, I noticed few things. First, I have a lot of bin folder with the same files. Second, that we are rebuilding the same file because they referenced them. To improve, I decided to edit all projects to output on the same bin folder. Third, the jobs folder that contains all scripts are heavy on the writing.

I decided to have every project to output in the bin folder of the website. The reason is that I have IIS taking the bin folder as source so every time I build I can just refresh the browser to get the website with the latest version without deploying. After that, I went into all projects references and click reference to other projects and changing the Copy Local to false.


From there, I cleaned up everything (all bin folder empty) and re-build everything to see how the performance improved. First, the Build Monitor extension is showing some improvements:

 - 00h 00m 00s 505ms	-- ComplexTypes --
 - 00h 00m 00s 601ms	-- ModelContracts --
 - 00h 00m 00s 352ms	-- ComplexTypesUnitTest --
 - 00h 00m 01s 453ms	-- MySqlToMsSql --
 - 00h 00m 01s 383ms	-- ValueObjects --
 - 00h 00m 03s 002ms	-- CrossLayer --
 - 00h 00m 01s 402ms	-- ComplexTypesWithValueObjects --
 - 00h 00m 01s 488ms	-- CrossLayerUnitTest --
 - 00h 00m 01s 940ms	-- TestHelpers --
 - 00h 00m 02s 924ms	-- Model --
 - 00h 00m 01s 637ms	-- ValueObjectsUnitTest --
 - 00h 00m 01s 806ms	-- ComplexTypesWithValueObjectsUnitTest --
 - 00h 00m 01s 184ms	-- ModelBuilders --
 - 00h 00m 03s 282ms	-- ViewModel --
 - 00h 00m 05s 198ms	-- ModelUnitTest --
 - 00h 00m 02s 993ms	-- Mapping --
 - 00h 00m 05s 746ms	-- ModelBuildersTest --
 - 00h 00m 03s 516ms	-- ViewModelBuilders --
 - 00h 00m 06s 564ms	-- DataAccess --
 - 00h 00m 00s 788ms	-- ViewModelUnitTest --
 - 00h 00m 03s 636ms	-- MappingUnitTest --
 - 00h 00m 03s 634ms	-- Services --
 - 00h 00m 04s 583ms	-- DataAccessMigration --
 - 00h 00m 04s 983ms	-- DataAccessUnitTest --
 - 00h 00m 04s 610ms	-- ServicesIntegrationTest --
 - 00h 00m 06s 222ms	-- IoC --
 - 00h 00m 06s 657ms	-- WebServicesCore --
 - 00h 00m 11s 568ms	-- ServicesUnitTest --
 - 00h 00m 04s 573ms	-- ScriptIndiceMarketPoint --
 - 00h 00m 06s 860ms	-- DataAccessIntegrationTest --
 - 00h 00m 04s 658ms	-- ScriptCleanExpiredOrders --
 - 00h 00m 04s 715ms	-- ScriptEndContests --
 - 00h 00m 05s 408ms	-- ScriptInitializeCache --
 - 00h 00m 05s 637ms	-- ScriptDailyPerformance --
 - 00h 00m 01s 181ms	-- ScriptBuyOrders --
 - 00h 00m 01s 760ms	-- ScriptSymbolChange --
 - 00h 00m 01s 831ms	-- ScriptBotRegisterContest --
 - 00h 00m 01s 770ms	-- ScriptFindUpcomingSplit --
 - 00h 00m 01s 766ms	-- ScriptFindUpcomingSymbolRename --
 - 00h 00m 07s 225ms	-- FrontEndSharingLayer --
 - 00h 00m 09s 518ms	-- ScriptEndOfDayBadges --
 - 00h 00m 09s 531ms	-- ScriptBuyShortOrders --
 - 00h 00m 09s 540ms	-- ScriptBotBuyContest --
 - 00h 00m 08s 867ms	-- ScriptUpdateCompanyInformation --
 - 00h 00m 11s 291ms	-- ScriptSymbolSplit --
 - 00h 00m 11s 304ms	-- ScriptSellShortOrders --
 - 00h 00m 10s 675ms	-- ScriptSellOrders --
 - 00h 00m 02s 717ms	-- ScriptInterdayStatisticPortefolios --
 - 00h 00m 10s 860ms	-- WebSite --
 - 00h 00m 03s 206ms	-- WebSiteUnitTest --
 - 00h 00m 03s 952ms	-- Architectures --
Time Elapsed: 00h 01m 32s 275ms 

The time to build is cut by half. This is already better. If we take the Process Monitor we can see the reason: we write only 51 megs.


Finally, if I unload all jobs project (scripts) one, I have a build time of 1m13sec. Not a huge improvement, but still 20 seconds less! The initial performance of 2 minutes 51 seconds to 1 minute 13 secconds is quite appreciable. With all these changes some problems raised. First, when pushing the code in the continuous integration (ci) environment, the build server is not able to build the whole solution. This is because the build server builds the startup project which doesn’t copy local any references. The second problem is when you deploy. Visual Studio Publish mechanism build the main project too, whence the same consequences. So, we need to add additional steps to build everything and we will come back to have some performance lost.

An other direction is to remove the most possible projects. This approach is fine but limited to what you can group together. For example, I have 1 web project, and about 14 web jobs. This mean a minimum of 15 projects. If we want to divide unit tests from the code, we can add 1 more project. If we want to share webjobs and website logic we add one more project. Still, we have half the number of project and while working on the shared tier and website, it is always possible to unload from the main solution every webjob projects. The best way to move everything is to create a shared project, that I called “ApplicationTier”. The website project remains the same but refers this new project. Inside Visual Studio, we need to go one by one in each project to drag-and-drop all files in a folder with the same name of the project. The final result is easy to read and consolidate a lot of project within one but with familiar structure. At the end, the result was very impressive. Instead of taking 2 minutes 51 seconds the build time was at 54 seconds.

 - 00h 00m 00s 775ms	-- MySqlToMsSql --
 - 00h 00m 06s 799ms	-- ApplicationTier --
 - 00h 00m 06s 064ms	-- DataAccessMigration --
 - 00h 00m 03s 400ms	-- ApplicationTierUnitTest --
 - 00h 00m 09s 714ms	-- WebServicesCore --
 - 00h 00m 11s 656ms	-- WebSite --
 - 00h 00m 02s 612ms	-- ScriptCleanExpiredOrders --
 - 00h 00m 02s 312ms	-- ScriptFindUpcomingSymbolRename --
 - 00h 00m 05s 217ms	-- ScriptEndContests --
 - 00h 00m 05s 271ms	-- ScriptBuyShortOrders --
 - 00h 00m 05s 345ms	-- ScriptBuyOrders --
 - 00h 00m 03s 508ms	-- ScriptFindUpcomingSplit --
 - 00h 00m 07s 308ms	-- ScriptDailyPerformance --
 - 00h 00m 08s 350ms	-- ScriptSymbolChange --
 - 00h 00m 08s 447ms	-- ScriptBotRegisterContest --
 - 00h 00m 03s 731ms	-- ScriptEndOfDayBadges --
 - 00h 00m 03s 327ms	-- ScriptSymbolSplit --
 - 00h 00m 04s 775ms	-- ScriptBotBuyContest --
 - 00h 00m 04s 728ms	-- ScriptUpdateCompanyInformation --
 - 00h 00m 02s 835ms	-- ScriptSellOrders --
 - 00h 00m 02s 907ms	-- ScriptSellShortOrders --
 - 00h 00m 01s 714ms	-- ScriptInterdayStatisticPortefolios --
 - 00h 00m 01s 608ms	-- ScriptInitializeCache --
 - 00h 00m 01s 579ms	-- ScriptIndiceMarketPoint --
 - 00h 00m 04s 239ms	-- Architectures --
 - 00h 00m 05s 405ms	-- WebSiteUnitTest --
[3] Time Elapsed: 00h 00m 54s 059ms 

By reducing the amount of project, we have a lot less references that needs to be cut. The number of megs written on disk is about 550 megs now. The main bottle neck is all the scripts for webjobs. Since all jobs are just entry points to the ApplicationTier, if they share the same bin folder, it will reduce by a lot the building time. This is because that the first script project to build will get the binary files in the bin folder, and subsequent scripts will just build the executable project without building again the references. The result is the following 31 seconds, mainly because only 196 megs go written on the disk.

 - 00h 00m 01s 633ms	-- MySqlToMsSql --
 - 00h 00m 06s 483ms	-- ApplicationTier --
 - 00h 00m 06s 696ms	-- DataAccessMigration --
 - 00h 00m 01s 458ms	-- ApplicationTierUnitTest --
 - 00h 00m 08s 610ms	-- WebServicesCore --
 - 00h 00m 09s 298ms	-- WebSite --
 - 00h 00m 01s 044ms	-- ScriptCleanExpiredOrders --
 - 00h 00m 00s 719ms	-- ScriptFindUpcomingSymbolRename --
 - 00h 00m 01s 140ms	-- ScriptFindUpcomingSplit --
 - 00h 00m 02s 132ms	-- ScriptDailyPerformance --
 - 00h 00m 02s 282ms	-- ScriptBuyOrders --
 - 00h 00m 02s 216ms	-- ScriptBotRegisterContest --
 - 00h 00m 02s 477ms	-- ScriptBuyShortOrders --
 - 00h 00m 02s 295ms	-- ScriptSymbolChange --
 - 00h 00m 00s 771ms	-- ScriptEndOfDayBadges --
 - 00h 00m 02s 619ms	-- ScriptEndContests --
 - 00h 00m 00s 811ms	-- ScriptUpdateCompanyInformation --
 - 00h 00m 00s 858ms	-- ScriptBotBuyContest --
 - 00h 00m 00s 764ms	-- ScriptSymbolSplit --
 - 00h 00m 03s 963ms	-- ScriptSellShortOrders --
 - 00h 00m 04s 022ms	-- ScriptSellOrders --
 - 00h 00m 03s 957ms	-- ScriptInitializeCache --
 - 00h 00m 04s 030ms	-- ScriptInterdayStatisticPortefolios --
 - 00h 00m 03s 683ms	-- ScriptIndiceMarketPoint --
 - 00h 00m 04s 649ms	-- WebSiteUnitTest --
 - 00h 00m 01s 533ms	-- Architectures --
Time Elapsed: 00h 00m 31s 471ms 

I quickly started the RamDisk tool to see if having the scripts bin folder could benefit of it and I haven’t see any improvement. Finally, I am pretty happy of the end result. I can always unload all scripts and this can be done easily since they are inside a folder. Inside Visual Studio, right click the folder that contain these folders and click “unload project”. This will unload all of them in 1 operation. For further optimization, we could also unload the migration project and unit tests and by doing so having less than 20 seconds of total build time.

How to automatic have your architecture validated by MsBuild

Visual Studio Ultimate edition has an architecture diagram that can be validated by right clicking the architecture. This is very interesting for project that become bigger and bigger to ensure that none of the developers by-pass layers or libraries in a way that is not intended.


The problem is that people tend not do validate the architecture for several reasons. The simplest one is that maybe not a lot of your developers has the Ultimate edition. Nevertheless, the good news is that Visual Studio Online lets you do what you can do locally with MsBuild. MsBuild with the argument /p:ValidateArchitecture=true will validate any architecture diagram you have.

msbuild /p:ValidateArchitecture=true

This can also be setup when building on the cloud. Open Visual Studio, go in the Builds tab from the Team Explorer. Edit any of your build definition, select Process and go under Build and Advanced. You can set from there MsBuild’s arguments. Use the same argument that you are using locally : /p:ValidateArchitecture=true.


When the architecture is not respected, the build server will thrown an error saying the problem within the architecture layer/code. This is the best way to ensure a strong architecture on the long term.