It's 2AM and I want to Sleep but IIS is Being Uncooperative

09 Jan, 2024

Last night I was ready to deploy a new application on our Windows Server running IIS. All of the files were ready to copy over. They worked completely fine when being accessed as a self-contained executable¹. Everything pointed to it being a smooth deployment and me being able to head off to bed until users logged on in the morning.

Using Visual Studio I published a framework-dependent version of the application and copied my files over to a directory on Windows Server. I decided to first deploy to the validation side of IIS to do a quick test making sure everything would still work.

Site1 worked great, no problems, I gained confidence! I load up Site2, and it refuses to talk to the new application. Site2 keeps complaining it can't read the output properly. That's strange, I can load it just fine if I directly access the application, and Site1 uses many of the same endpoints and it works.

The hours go by, I'm unable to figure out exactly why this isn't working on one validation site, but maybe Site2 is broken in validation². I opted to try a production version of my new application that the validation sites could talk to and see if perhaps there was just missing data. Once again, Site1 worked fine, Site2 had issues.

The clock turned to midnight at this point and no one is using the application, so I figured I'd try deploying to the production side of IIS and hope that Site2 in validation was broken somehow, but that production worked.

Now nothing works.

What??? The new application works completely fine as a self-contained application, works 75% of the way on IIS' validation site, and now works 0% on IIS' production site. What is going on here?

I continued to tweak and try new ideas until 2am. No matter what I did the site simply wouldn't work and the only error I got was 503. Frustratingly, nothing showed up on any of the logs. My only clue was a 503 error. I had enough by this point, sent an email to the team reporting my failure and went to bed³.

In the morning the team had a meeting to go over a few different ideas to tease out what was going on:

• Turn off a Middleware reporter (not needed for production)
• Disable the ASP.NET Core's internal HTTPS redirection since IIS will handle that
• Change how configuration was done in Program.cs
• On the production site in IIS, create a new Application and deploy the test to that to see if the same error is given there, and if so provide a way to troubleshoot during the day

The middleware didn't have an impact, however ASP.NET Core's internal HTTPS redirection app.UseHttpsRedirection(); in Program.cs which is a part of the template was what kept breaking Site2. Site2 only wanted to access via HTTP, and the HTTPS redirection was failing. Finally I had a fully featured application behind IIS, though only the validation side.

Since I know IIS can handle things, I created a new Application on the production site, pointed it to my code, and restarted the site. I opened a new browser window, entered the URL, and there it is

503

Plus side, I've got a model to test and play with to figure out why I'm getting a 503. Configuration was not the culprit making no difference after I applied changes. Still, there were no errors anywhere and even trying to increase my error reporting didn't help. All I had to go off of was a single 503.

Another hour goes by and I'm no closer. I've resorted to exporting each site from IIS and am manually combing through what could the differences be between production and validation. A team member reaches out in the group chat we made following this issue and says they opened the site in an InCognito window which yielded a more specific error

Not only does this error show the specific 503 instead of just HTTP ERROR 503, but it tells me exactly what's happening. I followed the instructions to create a new application pool, set the application to use that pool and bam, ✨ it just works! ✨

I don't know why IIS returns different errors if in a private browsing mode or not, or especially here why would one be so much more specific. Everyone else, trying in non-private browsers, only saw HTTP ERROR 503. This team member opted to try a private mode and found the key.

Looking back, there were a few other clues that should have tipped me off to differences, but being this is my first time dealing with deploying to IIS with an ASP.NET Core application and developing a full application in ASP.NET Core, you don't know what you don't know.

The breadcrumbs:

• The production site on IIS hosted additional applications compared to validation
• Logs were showing successful starts of the validation side, but was quiet for production
• The application pools were shared across sites
• I always started validation first when testing

Logs, or the lack thereof were the biggest indicator that my application was never actually being started. Then, ASP.NET Core works differently from previous iterations (talking Classic ASP) where the ASP.NET Core applications will run as separate applications, but heavily integrated with IIS. I assume this contributes to why Application Pools that previously felt endless now can only handle a single application, since they fundamentally run differently.

The default template for an ASP.NET Core web application also contributed to some of the issue, since by default it was pushing HTTPS when that wouldn't necessarily be needed by it, and without looking carefully into each and every line, a lot of the template felt like "this is what's required to start out" rather than "here's an opinionated start." It is worth noting that by recognizing it was a server configuration issue, the only part of the code I had to look at was the consolidated Program.cs, so if you can deduce the type of error you are having the C# structure can be beneficial.

The eagle-eyed among you may have noticed that I have something in common with Taylor Swift. It's me, hi, I'm the problem, it's me. I put two ASP.NET Core applications in one pool. But now they are like kiddie pools, too small for multiple big apps. I always started validation before production. Had I left validation off or done it in the opposite order, it would have worked last night.

Moving forward, we are going to have a single application pool per ASP.NET Core application, and know what logs to look for. As for right now, I'm going to go get some sleep. Goodnight.

Footnotes 🐾

#.net core #c sharp #iis #webdev