IIS: Automatically detect 500 (Internal Error) and Recover the Application Pool

Discussion in 'Windows Guides' started by Nimrod, May 5, 2014.

  1. Nimrod

    Nimrod Exotic Vendor

    Joined:
    Jun 1, 2007
    Messages:
    1,991
    Likes Received:
    533
    Location:
    London, United Kingdom
    IIS Web Server: Automatically detect 500 (Internal Error) and Recover the Application Pool


    There's nothing worse than when you first deploy your production IIS + PHP setup (or even ASP.NET) and after a few hours or days find your site is offline with Error 500 ‘Internal Server Error' causing you hours of downtime until you noticed it and reboot IIS. The cause of this is usually because the PHP Application Pool powering your site has experienced an error and died a painful death but for some reason IIS didn't detect this and automatically reboot your Application Pool.

    Internal Server Error.PNG

    You can spend hours, or weeks debugging the situation. Is it your configuration issue or is it an error in your application? I know it took us a lot of tweaking at Digiex to get it perfect and stable. However during this time your site is up and down like a yoyo, and if you rely on ad revenue and search engine traffic all this downtime can be dangerous to your bottom line.

    For reference, we eventually stabilized it by turning off Rapid-Fail Protection, enabling idle time outs for to 30 minutes on PHP processes, set processing time limits of 60 seconds and setting the FastCGI Standard error mode to TerminateProcess as errors are returned. However these settings will likely be different for your environment based on the type of scripting you use (PHP, ASP.NET etc), the amount of traffic and the scripts you use.

    However these results took 3 weeks of damaging downtime to become realized, and during this time we created a fantastic script that ran every 30 minutes via a scheduled task and checked the site is still online. If it wasn't, it took automated actions to check its offline for sure then bring it back online if it's still down by resetting IIS.

    The script runs via scheduled task set at an interval of your choosing.
    It has the following logic:
    Basically the script is designed to check the error code, but excuse the occasional error you may experience and ensure if it's going to do a full reset of your web server that it is indeed down.


    Download:

    If you would like to use the code, you can download it below:

    Download Digiex IIS 500 Error Detection and Recovery Script for IIS


    To customize the script to your needs:

    Once you have the script, I would recommend running it as a scheduled task as an Administrator (Administrator rights are required to run the IISreset.exe process).


    Run as a Scheduled Task

    To do so, go to Control Panel > Administration Tools > Task Schedular and select Create Task.

    It's important you set the following setting:

    When running the task, use the following user account (and select you're own or another Administrator level account)
    Run whether user is logged on or not.
    Run with highest privileges

    Scheduled Task.PNG

    Set it to run Daily, and repeat at a set time (recommended 30 minutes):

    Scheduled Task 2.PNG

    Set the Actions which is to start the bat file, as well as make sure the Start in field is filled with the full folder location of where the script is stored so that when it's running, it can find the curl.exe which is a required part of the script to allow it to check the HTTP Status Code

    Scheduled Task 3.PNG

    Done, the script is now installed and monitoring your web server. On the Last Run Result if it shows the result (0x0) it means it has run successfully.

    If you wish to monitor if the script has had any errors and reset IIS, you can check the Event Viewer and you will find notices like this whenever it has actioned any restarts for you in the System Log:

    Code:
    IIS stop command received from user DIGIEX-WEB\ADMINUSERTEST. The logged data is the status code.
    
    IIS start command received from user DIGIEX-WEB\ADMINUSERTEST. The logged data is the status code.
     

    Attached Files:

    Rick, InsaneNutter and xzKinGzxBuRnzx like this.

Share This Page