One notable project I did with .Net Core involved the development of a mobile app and web app for a large cinema chain. One of the requirements was that should be able to handle a large volume of traffic. Game on!
After going through the project's technical assessment, we decided to go with .Net core for the back-end services.
So we did the development work and when it came to do performance testing, something did not go right. We were expecting to at least exceed 500 requests per second easily but it was not meant to be.
Our AWS Fargate containers hosting the .Net core apps were dying at 50 requests per second. Even stranger is that CPU and memory consumption were all low. The apps were just not responding to requests.
What the heck is going on? It took us a couple of wasted days to figure things out and we discovered the bottlenecks were caused by three culprits.
Culprit #1: Not going async all the way
For any IO thread blocking methods, you will need to go async or else your app will slow to a crawl after a certain number of concurrent requests.
As a result, we had to go through every code that is an IO thread blocking code which were non- asynchronous and change them to be asynchronous. These changes were mostly in code that made requests to an external API or database queries via Entity Framework (EF).
One area of code which we frequently overlooked was that in EF core, they have methods like ToListAsync() and FirstOrDefaultAsync(). We learned the hard way that instead of using something like ToList(), to avoid performance hits, you will need to use ToListAsync() .
Not using async methods results in something called 'thread exhaustion' that will cause your .Net Core app to ground to a halt. The symptoms would be be low CPU and memory but the app takes a long time to respond or does not respond at all.
Culprit #2: Using WebClient instead of HttpClient
I've not investigated on why using WebClient instead of HttpClient would cause the .Net Core app to ground to a halt. Using WebClient, even when asynchronously, will cause performance bottlenecks. The moment, we ported our code to HttpClient, the app's performance exceeded expectations. It is something to keep note of and I will most likely look further into it in a separate post.
Culprit #3: Not using HttpClient properly
After fixing problems caused by culprit 1 and 2, most of our services successfully passed the performance tests. But there were a few services which were still failing. Turns out that it was due HttpClient being used incorrectly in certain parts of the code.
There is supposedly a tendency for developers to incorrectly use HttpClient. This consequently results in a an issue called port exhaustion. Like culprit #1, the symptoms would be be low CPU and low memory but the app is not responding.
Big thanks to this aspnetmonsters article for helping us troubleshoot the issue. The article is a must read.
My recommendation to fix this problem is to use HttpClientFactory when implementing HttpClient. I personally used this Microsoft documentation for my reference.
Thinking about it again, my colleagues and I are now suspecting that culprit#2 and culprit#3 are of the same nature.
So after rectifying all the above three issues, we did several more rounds of performance tests. And Bam! Now the apps able to execute more than 3000 requests per second. We were happy, clients are happy and so are the movie-going customers.
I hope this article will help those who encountered similar issues.
I am also planning to talk more about the architecture of the cinema project in a separate post as it is an interesting architecture with noteworthy pros and cons.