Hello, I've stumbled across two issues with the TRtcHttpClient component when I tried to connect to an URL through a proxy server.
Both issues can be reproduced with the RTC SDK Demo File_Client.exe.


Issue 1 - Specifying a non-existent proxy

Steps to reproduce in File Client Demo:
a) check the "Proxy" option
b) enter an address of a proxy that doesn't exist
c) enter an arbitrary proxy username and password
d) click the "Post" button

Result:
You should see that the GUI of the File Client Demo freezes, indicating that TRtcHttpClient isn't responsive anymore.
However this does not happen if you've skipped the above step c) and left the proxy username and password fields empty. In that case TRtcHttpClient properly calls the event OnConnectLost if it can't connect to the proxy.


Issue 2 - Getting a file from a redirected URL through a proxy

Steps to reproduce in File Client Demo:
a) check the "Proxy" option
b) enter a valid proxy address
c) enter a valid proxy username and password
d) enter a server address (URL) that is known to be redirected (e.g. www.ksta.de)
e) click the "Post" button

Result:
The response field of the File Client Demo shows the status "407 Proxy Authentication Required" even if the provided proxy credentials are definitely correct.
A trace of the HTTP request and response headers (using Wireshark e.g.) shows what happens:

GET http://www.ksta.de/ HTTP/1.1
HOST: www.ksta.de
Proxy-Connection: Keep-Alive
Cache-Control: no-cache
Pragma: no-cache

HTTP/1.0 407 Proxy Authentication Required
Proxy-Connection: Keep-Alive
Proxy-Authenticate: Basic realm="test"
----------------------------------------
GET http://www.ksta.de/ HTTP/1.1
HOST: www.ksta.de
Proxy-Connection: Keep-Alive
Cache-Control: no-cache
Pragma: no-cache
Proxy-Authorization: Basic dGVzdDp0ZXN0

HTTP/1.1 301 Moved Permanently
Date: Tue, 20 May 2008 08:28:33 GMT
Server: Apache
Location: http://www.ksta.de/shortcut.jsp?shortcut=
Content-Length: 249
Keep-Alive: timeout=10, max=500
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1
----------------------------------------
GET http://www.ksta.de/shortcut.jsp?shortcut= HTTP/1.1
HOST: www.ksta.de
Proxy-Connection: Keep-Alive
Cache-Control: no-cache
Pragma: no-cache

HTTP/1.0 407 Proxy Authentication Required
Proxy-Connection: Keep-Alive
Proxy-Authenticate: Basic realm="test"

Instead of stopping at the second response with "301 Moved Permanently" (as one would expect and as TRtcHttpClient does if no proxy is used) TRtcHttpClient sends an additional request to the redirected URL which leads to the weird 407 response.


Are these known issues? Is there an easy way to get around them with the current version 2.78 of the RTC SDK?

Regards,

Michael

Hi Michael,

Thank you for reporting these two issues.

Regarding Issue 1 ...

I am sorry, but I am unable to reproduce the problem you are describing in Issue 1. In my case, when I enter an URL of a WebServer as a proxy, I receive 404 Not Found as a result, the Client does not freeze. And if I enter something stupid (no real URL), the client is unable to post the request and returns with Request Aborted. In my case, the behavior is the same wether I have proxy username/password entered or leave it empty.

I have tested this on Windows Vista Business 32bit with the latest updates installed. Can you please debug the code to see where the code hangs and tell me which Windows version is causing the freezing problem?

The fastest way to find the location where the app freezes is to set breakpoints in the "rtcWinHttpCliProv.pas" unit at the beginning of following methods:
- WriteHeader(SendNow:boolean=True);
- SendHeaderOut(const s:string);
- Write(const s:string; SendNow:boolean=True);
- AcceptResponse;
- SetupProxy;

If there is a problem with the RTC SDK, it has to be in one of the above methods.

Regarding Issue nr. 2 ...

The behavior you are experiencing is either a problem of the WinHTTP API you are using, or a problem of the Proxy server which wants the client to authenticate for every single request. The correct behavior with the default options should actually be to receive the contents of the page where you are being redirected and not to get the 407 response.

I know that WinHTTP and WinInet APIs have automatic redirection set by default and had planned to make this behavior optional by adding a new parameter to the RtcHttpClient component, but the question remains why the WinHTTP API is not going through with the request but instead ends the request after the 407 response. If the problem is not in the WinHTTP API, then the only other problem source could be the proxy - which is asking for the user to authenticate AGAIN after he has already done so in the prior request.

I will be adding an option to disable auto-redirects when using WinHTTP and WinInet APIs, which should give you the 301 response you are expecting, but I can not release any updates for at least two more weeks because we are currently in the middle of porting the RTC SDK to FPC/Lazarus for Linux and the code is currently not production-ready.

If you can not wait so long, the only alternative would be for you to make the change in your version. The function you need to call from the WinHTTP api is already available ("InternetSetOption" function maps to the "WinHttpSetOption" API). Should you decided to add this option to your version of the RTC SDK (before we have the update ready), pleae read this article in the MSDN KB:
http://msdn.microsoft.com/en-us/library/aa384067(VS.85).aspx

As for the 1st issue, I will need your help to debug the code and let me know what you find out, since I am unable to reproduce the problem on my PC.

Best Regards,
Danijel Tkalcec

Hello Danijel,

thanks for your detailed answer. Let me first state that to my mind RDC SDK is a great piece of software with an extraordinary user support.

The two aforementioned issues occured on Windows XP Home and Professional, both with SP2.
I'm not sure if WinHTTP may be the source of the problems since I've unchecked the "WinHTTP" option in the File Client Demo.

For the first issue I've follwed your suggestion and set breakpoints in five Methods of TRtcWinHttpClientProvider. Here's the result: there's 1 call to WriteHeader, 1 call to SendHeaderOut and then SetupProxy is called again and again, causing the two remaining methods never get called. A deeper look shows that SetupProxy is called from the repeat loop in SendHeaderOut that never exits; in this loop there's the following code sequence

  else if not FHeaderOut then
    begin
    lastErr:=GetLastError;

the value of lastErr is always set to ERROR_WINHTTP_CANNOT_CONNECT leading to a jump to the SetupProxy call. As I can see now WinHTTP seems to be part of the (proxy) game even if UseWinHTTP is set to false.


For the second issue: I'm a bit sceptical that your suggestion to disable auto-redirection will work since I've disabled WinHTTP but I will give it a try anyway.

> If the problem is not in the WinHTTP API, then the only other problem source could be the proxy -
> which is asking for the user to authenticate AGAIN after he has already done so in the prior request.

But isn't this the normal behaviour of a proxy server? Could a proxy be sure that an unauthorized request is coming from the same client that has authorized itself before? A web browser always sends the first request to an URL w/o credentials but includes them with a second request after it has received a 407 response. See e.g. the following headers recorded with Firefox and the plugin Live HTTP Headers (the headers are anonymized and stripped down to the interesting parts):

GET / HTTP/1.1
Host: mycorp.com
Keep-Alive: 300
Proxy-Connection: keep-alive

HTTP/1.x 407 Proxy Authentication Required
Content-Type: text/html
Content-Length: 1149
Proxy-Authenticate: Basic realm="Internet MyCorp AG (http)"
Proxy-Connection: keep-alive
----------------------------------------------------------
GET / HTTP/1.1
Host: mycorp.com
Keep-Alive: 300
Proxy-Connection: keep-alive
Proxy-Authorization: Basic dGVzdDp0ZXN0

HTTP/1.x 302 Moved Temporarily
Location: https://mycorp.com/
Content-Length: 213
Content-Type: text/html; charset=iso-8859-1
Proxy-Connection: keep-alive
----------------------------------------------------------
GET / HTTP/1.1
Host: mycorp.com
Keep-Alive: 300
Connection: keep-alive

HTTP/1.x 407 Proxy Authentication Required
Content-Type: text/html
Content-Length: 1138
Proxy-Authenticate: Basic realm="Internet MyCorp AG (https)"
Proxy-Connection: keep-alive
----------------------------------------------------------
GET / HTTP/1.1
Host: mycorp.com
Keep-Alive: 300
Connection: keep-alive
Proxy-Authorization: Basic dGVzdDp0ZXN0

HTTP/1.x 200 OK
Content-Type: text/html
Content-Length: 629
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
----------------------------------------------------------


I will report the result of my attempt to solve issue #2 by disabling auto-reconnection.


BTW: I'm not a big fan of using WinHTTP. If you are developing a WinHTTP replacement for the FPC/Linux port of the RTC SDK, is there a chance that this code will be overtaken to the Windows version to let it become independent of WinHTTP?


Kind regards

Michael

Hello Danijel,
It seems that your suggestions have put me in the right direction.
After I changed line #1408 in rtcWinHttpCliProv.pas from

  else if not SetupProxy then

to

  else if (lastErr = ERROR_WINHTTP_CANNOT_CONNECT) or (not SetupProxy) then

the GUI freeze is gone and the OnConnectionLost event of TRtcHttpClient is fired instead
-> ISSUE 1 SOLVED!

Adding the lines

  winHttpDisableFeature := WINHTTP_DISABLE_REDIRECTS;
  InternetSetOption(hRequest, WINHTTP_OPTION_DISABLE_FEATURE,
                    Addr(winHttpDisableFeature), SizeOf(winHttpDisableFeature));

in rtcWinHttpCliProv.pas right before the call to SetupProxy in line #1496 make TRtcHttpClient receive the redirection response that my application can handle
-> ISSUE 2 SOLVED!

Thank you for your help.


Kind regards

Michael

Hi Michael,

Thank you very much for going the extra mile and solving the two issues.

As a little token of appreciacion, I have updated your RTC account to give you unlimited access to the RTC SDK Professional (when it is released). RTC SDK Professional will be an extension to the Open Source RTC SDK edition and implement some new features which will not be included in the Open Source edition of the RTC SDK.

For the issue nr. 1, I would recommend adding a private boolean variable "proxyOK" which will be set to FALSE before the loop, then changing the code at the current line #1408 to ...

      else
        begin
        if proxyOK then
          Break
        else
          begin
          proxyOK:=True;
          if not SetupProxy then Break;
          end;
        end;

This will make sure that all problems (not only connect errors) which can not be solved by re-setting proxy options and reposting the header will make the error surface (request aborted) instead of generating an endless loop.

Best Regards,
Danijel Tkalcec

Hello Danijel,

> As a little token of appreciacion, I have updated your RTC account to
> give you unlimited access to the RTC SDK Professional (when it is released).

Great! Thank you very much. I'm curious about the new features that the PRO version will bring.

> For the issue nr. 1, I would recommend adding a private integer variable "proxyOK"
> which will be set to 0 before the loop, and then changing the code at the current
> line #1408 to ...

> This will make sure that all problems (not only connect errors) which can not be solved by
> reposting the header up to 3 times (2 redirects) will make the error surface (request aborted)
> instead of generating an endless loop.

That's an important hint. I will modify my code accordingly.


Kind regards

Michael

Hi Michael,

I have decided to make the changes in the version I have prior to our FPC/Lazarus release and make the update available now, since the issue Nr. 1 is a serious bug which I can't leave hanging for weeks. This update I am releasing now also includes two other fixes, so I'd recommend you to download the new version 2.81, which is now available for download from the RTC Forums download area.

I am going to make the same update available on SourceForge today.

Best Regards,
Danijel Tkalcec

Hello Danijel,

I just downloaded the new version 2.81 and recompiled the File Client Demo. Unfortunately the GUI freeze that I reported with the first post is still there.

A quick investigation showed that line #1415 in rtcWinHttpCliProv.pas should be

  proxyOK:=True

instead of

  proxyOK:=False

since proxyOK:=False is the default anyway.


Kind Regards

Michael

Thank you for reporting this bug.

I have fixed it and uploaded it as RealThinClient SDK 2.82 (check RTC Download area). I hope there are no other problems with this release, but I'd like to ask you to check the new version and let me know if it works for you.

Best Regards,
Danijel Tkalcec

Hello Danijel,

Version 2.82 is working for me.


Kind regards,

Michael

Perfect!
Thank you for your help and your feedback, Michael :)

Best Regards,
Danijel Tkalcec