Using dotnet-counters for analyzing performance of the asp.net core application

This tool is available as nuget package and can be used to watch the performance of CPU and threadpool queue length (to check thread pool starvation) when requests are hitting your application.

I tested the performance of the application under load test using JMeter. I could see CPU usage very low(almost at zero for a much longer period) and high threadpool queue length which indicates thread pool starvation is happening. So we refactored the code with asynchronous APIs using async await and found the ThreadPoolQueue length is reduced to zero except at the start time and now CPU goes also up as threads are ready for processing by the CPU. We also analysed the dumps file and found many SqlConnectionWait and refactored the code for caching in such areas as those calls can be cached and no need to hit the DB for requests which are having same parameters. This further increased the application throughput value.

Listing down the steps to use this tool.

1. Install the tool as nuget package, refer https://docs.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-counters  

2. Run your asp.net core application with below command by going to the application path

dotnet run

Now the application will start running.


Once the application is up we have to run a load test, it can be done manually or via JMeter, I have used JMeter for hitting the application. Before running the load test, type this command in command prompt which will show the processID of the application

3. dotnet-counters ps //will show process-id

In the command output we can identify the process id of our application, now monitor the application with below command

4. dotnet-counters monitor --process-id 48084 --counters Microsoft.AspNetCore.Hosting[],System.Runtime[]

here as an example of process-id 48084

This will show up command window with showing the current CPU usage, no.of requests getting processed by the app and thread pool information etc.

Now choose a point where you see high CPU or high threadpool queue length is found, at this point we can take dumps with below command

5. dotnet-dump collect --process-id 48084 --type full

This will generate a dump file in a format like this: dump_20220425_155317.dmp

To load debug symbols for this dump so that debugging is possible in the code file.

6. dotnet-symbol --debugging --host-only dump_20220425_155317.dmp 

To analyse the dump file we can open the file with visual studio and run in Debug Managed Code, this will show up the parallel stack window which shows the stacks executed at the point of time we have taken the dump. From each stack we will get an idea where the time is lost and the values in the methods.

Besides this there is another command which gives insight about the CLR threads and GC details in a quick manner from command prompt itself

dotnet-dump analyze dump_20220425_155317.dmp //dump file example: dump_20220425_155317.dmp

which further prompts to enter a value what type of threads we need to analyse in the dump, we are intersted in clrthreads(i.e. managed threads only)

clrthreads

It will show all the managed threads at that time.