Logging issue with X-Forwarded-For, wrong location

Question

User-1714698417 posted

I have a really strange issue with logging the x-forwarded-for header in IIS logs on a few particular servers.

The log looks like this:

#Software: Microsoft Internet Information Services 8.5
#Version: 1.0
#Date: 2021-05-04 00:00:12
#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken
2021-04-20 11:24:51 2000000a3+X-Forwarded-For+-+ serverxxx x.x.x.x GET /path/path/paths.png - 443 - x.x.x.x HTTP/1.1 Mozilla/5.0+(Windows+NT+6.2;+WOW64)+AppleWebKit/538.1+(KHTML,+like+Gecko)+PhantomJS/2.1.1+Safari/538.1 ASPSESSIONIDSXXXXX=XXXXXX https://website.company.net/path/path/paths.asp sitename.company.net 200 0 0 51636 465 45

As you can see, the x-forwarded-for info looks all kinds of weird and it appears after the date/time. I have 5 x Windows Server 2012 R2 IIS servers which all have the problem. I have another ~50 servers which are absolutely fine. On the 50 servers that are OK, the x-forwarded-for field appears in the #fields of the log and correctly as the last entry of a line.

I have the logging field setup uniformly across all the servers on the IIS root as it was done via a script.

The IIS log file name doesn't get the _x at the end of the filename like the other servers do.

I can see in the applicationHost.config we have the following, which is exactly the same between a working server and a non working server. There are no other entries in there for <LogFile>

            <siteDefaults>
                <logFile logExtFileFlags="Date, Time, ClientIP, UserName, SiteName, ComputerName, ServerIP, Method, UriStem, UriQuery, HttpStatus, Win32Status, BytesSent, BytesRecv, TimeTaken, ServerPort, UserAgent, Cookie, Referer, ProtocolVersion, Host, HttpSubStatus" logFormat="W3C" directory="%SystemDrive%\inetpub\logs\LogFiles">
                    <customFields>
                        <clear />
                        <add logFieldName="X-Forwarded-For" sourceName="X-Forwarded-For" sourceType="RequestHeader" />
                    </customFields>
                </logFile>
                <traceFailedRequestsLogging directory="%SystemDrive%\inetpub\logs\FailedReqLogFiles" />
            </siteDefaults>

Does anyone have any idea why this is happening?

Answer 1

User1065476709 posted

Hi IISUserLondon,

As you can see, the x-forwarded-for info looks all kinds of weird and it appears after the date/time. I have 5 x Windows Server 2012 R2 IIS servers which all have the problem. I have another ~50 servers which are absolutely fine. On the 50 servers that are OK, the x-forwarded-for field appears in the #fields of the log and correctly as the last entry of a line.

Have you checked whether the corresponding log file in logFiels is correct? or you can try to reset iis and try again.

Right-click on cmd.exe and select run as administrator, at the command prompt, type IISRESET, press Enter.

Best regards,

Sam

Answer 2

User-1714698417 posted

Have you checked whether the corresponding log file in logFiels is correct? or you can try to reset iis and try again.

 

Unfortunately I'm not quite sure what you mean.

A working server has the following headers in the log and the log file itself has _x appended to the filename.

#Software: Microsoft Internet Information Services 8.5
#Version: 1.0
#Date: 2021-04-25 06:36:35
#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken X-Forwarded-For

The broken servers do not have the _x appended to the log filename, nor do they have "X-Forwarded-For" in the log header field (#fields)

Re the iis restart - we've restarted these server many times since the log fields were added to IIS.

Answer 3

User1065476709 posted

Hi IISUserLondon,

The broken servers do not have the _x appended to the log filename, nor do they have "X-Forwarded-For" in the log header field (#fields)

I tested X-Forwarded-For in my iis logs and it works normally. Please check your logFieldName and source spelling for errors.

Or you can also try to add X-Forwarded-For header  by PowerShell command at the server level.

Add-WebConfigurationProperty -pspath 'MACHINE/WEBROOT/APPHOST'  -filter "system.applicationHost/sites/siteDefaults/logFile/customFields" -name "." -value @{logFieldName='X-Forwarded-For';sourceName='X-Forwarded-For';sourceType='RequestHeader'}

Best regards,

Sam

Answer 4

User-1714698417 posted

There are definitely no spelling errors and that's the exact piece of PowerShell I used to set up all ~50 servers, including the ones with the issue.

Answer 5

User1065476709 posted

Hi IISUserLondon,

There are definitely no spelling errors and that's the exact piece of PowerShell I used to set up all ~50 servers, including the ones with the issue.

Have you tried using PowerShell commands in broken servers? Have you compared whether the configuration of this part of the web.config file between the two servers is the same?

Best regards,

Sam

Answer 6

User-1714698417 posted

Yes, I used the powershell command to setup the broken servers.

I've compared the applicationHost.config (as seen in the first post), which has identical logFile settings - I've compared with Notepad++. The section is identical.

Answer 7

User1065476709 posted

Hi IISUserLondon,

I've compared the applicationHost.config (as seen in the first post), which has identical logFile settings - I've compared with Notepad++. The section is identical.

Is your log folder correct? The number after W3SVC corresponds to your website id.

Best regards,

Sam

Answer 8

User-1714698417 posted

Yes the folders are correctly named as you describe. They are auto created by IIS itself.

Thanks

Answer 9

User1065476709 posted

Hi IISUserLondon,

Yes the folders are correctly named as you describe. They are auto created by IIS itself.

Have you tried other custom fields in broken servers?

Best regards,

Sam