Posts: 1
Threads: 1
Joined: Aug 2019
Reputation:
0
Location: Budapest
08-15-2019, 06:51 AM
(This post was last modified: 08-15-2019, 11:31 AM by Jose Nilton Pace.)
Hi,
any help would be useful.
This Visa server is quite specific (I have tried two commercial FTP clients /WS_FTP and TurboFTO/, none of them could connect to this server), two examples:
- requires a special sequence of FTP command during connection startup ( I could follow this making some modifications in IdFTP.pas)
- behaves a little bit special when using the CCC option (I'm stuck with this)
Server documentation states:
Quote:The client is expected to initiate the TLS shutdown after receiving the 200 response to the CCC request.
When I issue the CCC command, it receives the correct answer from the server, but after this the connection is broken. It seems the Indy Ftp client / OpenSSL solution handles this situation a little bit different as the server requires it, but I do not know how to change this.
ANY HELP WOULD BE WELCOME!
What I can see in the IdFtp.pas, after issuing the "CCC" command and receiving the acceptence from the server, the following is executed:
Code: (IOHandler as TIdSSLIOHandlerSocketBase).PassThrough := True;
And that's all!!!
I do not know how could any change be applied to modify the client behaviour.
Thanks in advance.
Tibor Peak
Posts: 652
Threads: 2
Joined: Mar 2018
Reputation:
35
Location: California, USA
08-16-2019, 09:43 PM
(This post was last modified: 08-16-2019, 10:03 PM by rlebeau.)
(08-15-2019, 06:51 AM)peak.tibor@gmail.com Wrote: Server documentation states:
Quote:The client is expected to initiate the TLS shutdown after receiving the 200 response to the CCC request.
When I issue the CCC command, it receives the correct answer from the server, but after this the connection is broken. It seems the Indy Ftp client / OpenSSL solution handles this situation a little bit different as the server requires it, but I do not know how to change this.
TIdFTP automatically issues 'CCC' on the first transfer after FTPS is activated and TIdFTP.UseCCC is true. But, when Indy "clears" a secure command connection to downgrade it back to unsecure, it doesn't actually perform any kind of TLS shutdown at all, it merely flags the SSLIOHandler to stop encrypting/decrypting subsequent data on the socket. The TLS session is still active, just ignored. The only way to shutdown the TLS session without closing the socket is to call OpenSSL's SSL_shutdown() function directly.
Per OpenSSL's documentation (emphasis is mine):
Quote:SSL_shutdown() tries to send the "close notify" shutdown alert to the peer. Whether the operation succeeds or not, the SSL_SENT_SHUTDOWN flag is set and a currently open session is considered closed and good and will be kept in the session cache for further reuse.
Note that SSL_shutdown() must not be called if a previous fatal error has occurred on a connection i.e. if SSL_get_error() has returned SSL_ERROR_SYSCALL or SSL_ERROR_SSL.
The shutdown procedure consists of 2 steps: the sending of the "close notify" shutdown alert and the reception of the peer's "close notify" shutdown alert. According to the TLS standard, it is acceptable for an application to only send its shutdown alert and then close the underlying connection without waiting for the peer's response (this way resources can be saved, as the process can already terminate or serve another connection). When the underlying connection shall be used for more communications, the complete shutdown procedure (bidirectional "close notify" alerts) must be performed, so that the peers stay synchronized.
SSL_shutdown() supports both uni- and bidirectional shutdown by its 2 step behaviour.
When the application is the first party to send the "close notify" alert, SSL_shutdown() will only send the alert and then set the SSL_SENT_SHUTDOWN flag (so that the session is considered good and will be kept in cache). SSL_shutdown() will then return with 0. If a unidirectional shutdown is enough (the underlying connection shall be closed anyway), this first call to SSL_shutdown() is sufficient. In order to complete the bidirectional shutdown handshake, SSL_shutdown() must be called again. The second call will make SSL_shutdown() wait for the peer's "close notify" shutdown alert. On success, the second call to SSL_shutdown() will return with 1.
If the peer already sent the "close notify" alert and it was already processed implicitly inside another function (SSL_read(3)), the SSL_RECEIVED_SHUTDOWN flag is set. SSL_shutdown() will send the "close notify" alert, set the SSL_SENT_SHUTDOWN flag and will immediately return with 1. Whether SSL_RECEIVED_SHUTDOWN is already set can be checked using the SSL_get_shutdown() (see also SSL_set_shutdown(3) call.
It is therefore recommended, to check the return value of SSL_shutdown() and call SSL_shutdown() again, if the bidirectional shutdown is not yet complete (return value of the first call is 0). As the shutdown is not specially handled in the SSLv2 protocol, SSL_shutdown() will succeed on the first call.
(08-15-2019, 06:51 AM)peak.tibor@gmail.com Wrote: What I can see in the IdFtp.pas, after issuing the "CCC" command and receiving the acceptence from the server, the following is executed:
Code: (IOHandler as TIdSSLIOHandlerSocketBase).PassThrough := True;
And that's all!!!
Yup. Setting PassThrough to True does not perform an actual TLS shutdown:
Code: procedure TIdSSLIOHandlerSocketOpenSSL.SetPassThrough(const Value: Boolean);
begin
if fPassThrough <> Value then begin
if not Value then begin
// issues a TLS handshake here ...
else
// merely resets the socket buffer sizes here ...
end;
fPassThrough := Value;
end;
end;
(08-15-2019, 06:51 AM)peak.tibor@gmail.com Wrote: I do not know how could any change be applied to modify the client behaviour.
You would have to modify Indy so that immediately after 'CCC' succeeds, SSL_shutdown() is called until "close notify" events have been sent and received. Doing that in TIdSSLIOHandlerSocketOpenSSL.SetPassThrough() would be best. Only a few places in Indy ever set PassThrough to True after SSL/TLS has been activated, and most of those are in TIdFTP and TIdFTPServer for 'CCC' and 'REIN' handling. So I think it should be safe to handle SSL_shutdown() there.
Try this:
Code: procedure TIdSSLIOHandlerSocketOpenSSL.SetPassThrough(const Value: Boolean);
begin
if fPassThrough <> Value then begin
if not Value then begin
if BindingAllocated then begin
if Assigned(fSSLContext) then begin
OpenEncodedConnection;
end else begin
raise EIdOSSLCouldNotLoadSSLLibrary.Create(RSOSSLCouldNotLoadSSLLibrary);
end;
end;
end
else begin
if fSSLSocket <> nil then begin
if SSL_shutdown(fSSLSocket.fSSL) = 0 then begin
SSL_shutdown(fSSLSocket.fSSL);
end;
end;
{$IFDEF WIN32_OR_WIN64}
// begin bug fix
if BindingAllocated and IndyCheckWindowsVersion(6) then
begin
// disables Vista+ SSL_Read and SSL_Write timeout fix
Binding.SetSockOpt(Id_SOL_SOCKET, Id_SO_RCVTIMEO, 0);
Binding.SetSockOpt(Id_SOL_SOCKET, Id_SO_SNDTIMEO, 0);
end;
// end bug fix
{$ENDIF}
end;
fPassThrough := Value;
end;
end;
Posts: 4
Threads: 0
Joined: Aug 2019
Reputation:
0
Location: Budapest
Hi rlebeau,
thanks for advising.
I made that modification, using a SVN version 5515 of Indy.
It really makes the SSL shutdown now, I can see the 'close notify' TLS messages in the WireShark log.
But it fails when I try to get a file from the server with "Access violation at address 00407E8E in module 'IdFtp_Tests_2.exe'. Read of address 00000020" message that appears after a while (destination local file is created, but empty).
I cannot really debug the program, before I cannot access the server from my developer pc, I can write only log messages to see at which step the program is executing.
It seems to me the problem is in "TIdFTP.InitDataChannel" where "PassThrough" is set to true for the newly cloned IOHandler of the datachannel, and it causes the SSL_shutdown to be called for that IOHandler.
Can you advise how to proceed?
Thank you in advance.
Tibor
Posts: 652
Threads: 2
Joined: Mar 2018
Reputation:
35
Location: California, USA
(08-18-2019, 04:08 PM)PeakTibor Wrote: I made that modification, using a SVN version 5515 of Indy.
It really makes the SSL shutdown now, I can see the 'close notify' TLS messages in the WireShark log.
Good.
(08-18-2019, 04:08 PM)PeakTibor Wrote: But it fails when I try to get a file from the server with "Access violation at address 00407E8E in module 'IdFtp_Tests_2.exe'. Read of address 00000020" message that appears after a while (destination local file is created, but empty).
An AV at a memory address near 0 usually indicates a nil pointer is being accessed. What does the call stack look like when the AV is raised? Which module inside your EXE does memory address $00407E8E reside in?
(08-18-2019, 04:08 PM)PeakTibor Wrote: I cannot really debug the program, before I cannot access the server from my developer pc, I can write only log messages to see at which step the program is executing.
Well, then you will likely have to install a 3rd party exception logger into your app, like MadExcept, EurekaLog, JclDebug, etc so that you can log those kind of details when the exception happens.
(08-18-2019, 04:08 PM)PeakTibor Wrote: It seems to me the problem is in "TIdFTP.InitDataChannel" where "PassThrough" is set to true for the newly cloned IOHandler of the datachannel, and it causes the SSL_shutdown to be called for that IOHandler.
Try changing the following line in SetPassThrough() from this:
Code: if fSSLSocket <> nil then begin
To this:
Code: if (fSSLSocket <> nil) and (fSSLSocket.fSSL <> nil) then begin
Something else I notice is that the fPassThrough member in IdSSL.pas is initialized to False by default. It probably should be initialized to True instead. Many places inside of Indy force PassThrough=True as soon as they create new instances of SSLIOHandler objects.
Posts: 4
Threads: 0
Joined: Aug 2019
Reputation:
0
Location: Budapest
Hi rlebeau,
thanks for all your help.
Finally I made some additional changes:
1. I created a separate "ShutDownSSL" procedure, the same was as Indy has a "StartSSL"procedure, and I call explicitly this new procedure after issuing the "CCC" command.
2. I created a separate procedure to issue the CCC ("SendCCC") and removed this part from "SendDataSettings".
This specific VISA server does not accept the original sequence of commands in "SendDataSettings": SendPBSZ -> SendPROT -> CCC
And I think it may have right. PBSZ and PROT should/could be set before login, CCC can be issued only after login, because sending userid and pwd in clear command channel is usually not acceptable.
So, before login I call SendDataSettings, after login I call SendCCC, that sets PassThrough := True and calls ShutDownSSL.
Maybe my changes are not generally appliable, but for my specific situation this is what is required.
Thanks again
Tibor
Posts: 652
Threads: 2
Joined: Mar 2018
Reputation:
35
Location: California, USA
I have just now checked in some changes for TIdIOHandlerSocketOpenSSL, more inline with the changes I described earlier rather than the changes you mention above, though.
(08-26-2019, 06:40 AM)PeakTibor Wrote: This specific VISA server does not accept the original sequence of commands in "SendDataSettings": SendPBSZ -> SendPROT -> CCC
And I think it may have right. PBSZ and PROT should/could be set before login, CCC can be issued only after login
Are you sure about that? According to RFC 2228, PBSZ "must be preceded by a successful security data exchange", same as CCC (PROT "must be preceded by a successful protection buffer size negotiation"). That implies that none of these commands can be sent until after a "successful security data exchange", which is performed using the ADAT command after an AUTH command, but TIdFTP does not currently support/use ADAT.
I haven't made any changes to TIdFTP yet, I'll have to look into that later. Feel free to open a ticket at https://github.com/IndySockets/Indy/issues so I don't forget about it.
Posts: 4
Threads: 0
Joined: Aug 2019
Reputation:
0
Location: Budapest
Hi rlebeau,
I may missing something, so I try get help.
The requested sequnce of FTP commands/activities with this Visa server is as follows:
- AUTH TLS
- TLS handshake
- PBSZ 0
- PROT P (Data channel must be encrypted)
- USER, PASS
- CCC (if Clear Command Channel have to be used)
Is it possible with the current idFTP solution to follow this?
Thanks in advance
Tibor
Posts: 652
Threads: 2
Joined: Mar 2018
Reputation:
35
Location: California, USA
08-30-2019, 04:45 PM
(This post was last modified: 08-30-2019, 04:51 PM by rlebeau.)
(08-30-2019, 05:50 AM)PeakTibor Wrote: The requested sequnce of FTP commands/activities with this Visa server is as follows:
- AUTH TLS
- TLS handshake
- PBSZ 0
- PROT P (Data channel must be encrypted)
- USER, PASS
- CCC (if Clear Command Channel have to be used)
Is it possible with the current idFTP solution to follow this?
Possible, but not in a straight-forward manner.
TIdFTP does have SendPBSZ() and SendPROT() methods, which you can call whenever you want. However, they are protected methods, so you would need to use a descendant/accessor class to reach them, as well as to set TIdFTP.FDataSettingsSent=True so that TIdFTP.SendDataSettings() does not send its own PBSZ/ PROT/ CCC commands.
The TIdFTP.Login() method does not natively allow you to send custom commands after it sends AUTH and performs the TLS handshake, and before it sends USER/ PASS/ ACCT, so you would have to handle the login sequence manually. You could set TIdFTP.AutoLogin=False and handle the entire login sequence manually. Or, you could set TIdFTP.ProxySettings.ProxyType=fpcmCustomProxy and use the TIdFTP.OnCustomFTPProxy event to send PBSZ, PROT, and USER/ PASS after TIdFTP.Login() sends AUTH and performs the TLS handshake for you.
That just leaves CCC, which you can disable by setting TIdFTP.UseCCC=False and then send it manually when needed.
Posts: 4
Threads: 0
Joined: Aug 2019
Reputation:
0
Location: Budapest
Hi rlebeau,
thanks for advising.
I think we have to consider this Visa server requirements as NON-STANDARD, so there is no need to build its support into the idFtp source.
As I described above, I could manage to modify idFtp source to support this Visa server, but it should not be considered as general, so I will not open an ISSUE ticket.
Thanks again for your support.
Tibor
|