**************** FTPS Extender Version 44001 Released ***********

[td]

The FTPS extender provides WinBatch® scripts File Transport Protocol Secure functionality. FTPS uses command and data channel encryption to protect passwords and data during client/server file transfers.

It is a bit unusual but we decided to turn an internal-use extender into something that could be released to WinBatch users. The conversion mostly involved writing documentation and integrating the extender fully into the WinBatch/WinBatch Studio* environment.

Given its origins, the extender lacks the automagic features so popular with some and the documentation is a little basic. It has a collection of 13 functions that work more or less as their names would indicate.

The online version of the documentation can be found here:

FTPS Extender Docs
The extender table of contents appears when you click on the "show" link in the upper-left-hand corner.

Customers with a current license can download the extender here:

FTPS Extender Download Link

Or you can use the vcheck script.

* Some aspects of WinBatch Studio integration will not take effect until the next WinBatch+Compiler release but this does not limit the extender's functionality in any way.

[spl]

Nice addition. Did notice
The FTPS extender cannot...
Provide upload or download progress information
Issue ad hoc FTP commands.
Transfer data using the SFTP protocol

bummer on lack of SFTP. I had gotten quite comfortable with WinSCP .NET uploads and downloads.

 

[kdmoyers]

This looks pretty cool Tony.  I have a business partner that publishes prices on an FTPS server, so I will use this if we hook up to get that data on the regular.  Thanks!!
 
((Around here on the LAN we use SFTP for simpler firewall config and no cert needed.))

[td]

Nice addition. Did notice
The FTPS extender cannot...
Provide upload or download progress information
Issue ad hoc FTP commands.
Transfer data using the SFTP protocol

bummer on lack of SFTP. I had gotten quite comfortable with WinSCP .NET uploads and downloads.

As mentioned it is an internal use extender and ILC doesn't need SFTP.  If you like WinSCP .NET then you should use it. It did not match ILC's needs.

[td]

This looks pretty cool Tony.  I have a business partner that publishes prices on an FTPS server, so I will use this if we hook up to get that data on the regular.  Thanks!!
 
((Around here on the LAN we use SFTP for simpler firewall config and no cert needed.))

Never needed SFTP because all ILC firewalls auto-configure for explicit FTPS when detected.  The SSH part of SFTP still needs a certificate from somewhere even if it is self-signed.

Should add that the remote FTP servers need the addition of a data port range as part of the FTP server configuration. It is just part of the FTP server service/daemon setup process. ILC's VPS server firewall, the VPS provider's firewall, and Cloudflare auto-configure for the port range.

[spl]

If you like WinSCP .NET then you should use it. It did not match ILC's needs.

Yep. Pick want you want to deal with and how to deal with...

[td]

We assumed when we decided to publish it that there wasn't much interest in FTP. If that was an erroneous assumption, we would consider adding SFTP to the extender. 

[spl]

NP. I did test the extender on the site in the code below. It supports (according to authors) both FTPS and SFTP. I tested the connection and file list for FTPS with both plain text and encrypted credentials based on code from another author. I then tested in both 32 and 64 bit with the Extender. I could connect with both but got a strange error when trying to list files. Then I switched the protocol to unencrypted and worked fine. The protocol for the server is, I believe, TLS12. You can comment/uncomment to test in the attached. Would there be a way to determine the configuration of an FTP server prior to connecting?

Code Select Expand

AddExtender('ilcfs44i.dll',0,'ilcfs64i.dll')
host = 'test.rebex.net'
port = 21
;port = 990
user = 'demo'
pwd  = 'password'

;no error on connect, but error on fsDirList
client = fsClientInit(host, port, user, pwd)

;comment above and uncomment and fsDirList returns files
;client = fsClientInit(host, port, user, pwd, @fsFTP)

dir = "pub/example"
list = fsDirList(client, dir, @fsList)
file = AskItemlist('Select a File', list, @lf, @sorted, @single)
Message('You Selected',file)
fsClientDelete(client)
Exit


[spl]

Correction to last post. The failure was on Win11 but script worked on Win10 [as compiled 32bit exe, with default parameters]. I added @false to the optional names_only parameter. It does give information but wish it could be reduced to specific attributes. Attached json in .zip for specifics [should be .csv, but I am having fun creating json].

[spl]

Another Win11 test and error associated with default parameters, but will list files adding @fsFTP as final parameter. As with previous post where script failed on Win11 but worked on Win10 this may fall into same scenario. Will plan to compile and run on Win10.

Code Select Expand

AddExtender('ilcfs44i.dll',0,'ilcfs64i.dll')
host = 'ftp.dlptest.com'
port = 21
;port = 990
user = 'dlpuser'
pwd  = 'rNrKYTX9g7z3RgJRmxWuGHbeu'

;no error on connect, but error on fsDirList
client = fsClientInit(host, port, user, pwd)

;comment above and uncomment and fsDirList returns files
;client = fsClientInit(host, port, user, pwd, @fsFTP)

dir = "/"
list = fsDirList(client, dir, @fsList)
file = AskItemlist('Select a File', list, @lf, @sorted, @single)
Message('You Selected',file)
fsClientDelete(client)
Exit


[td]

Correction to last post. The failure was on Win11 but script worked on Win10 [as compiled 32bit exe, with default parameters]. I added @false to the optional names_only parameter. It does give information but wish it could be reduced to specific attributes. Attached json in .zip for specifics [should be .csv, but I am having fun creating json].

The server determines the specific attributes that the extender returns. For example, if the FTP server is Windows IIS, the results are usually a subset of cmd.exe's "dir /a" command. The content is usually a subset of the "ls -l" shell command if the FTP server is a common Linux FTP server. But it could be anything based on the FTP server service/daemon coding.

[td]

Another Win11 test and error associated with default parameters, but will list files adding @fsFTP as final parameter. As with previous post where script failed on Win11 but worked on Win10 this may fall into same scenario. Will plan to compile and run on Win10.

The error results from the extender being unable to verify the server certificate. It does this by working up the certificate chain to the root certificate. There can be several causes for verification failure. The most common is that the certificate's root or intermediate certificates are not in the Windows certificate store. Like most other Windows-based applications the extender uses the Windows default certificate store.

The reason it works on Windows 10 and not Windows 11 could be because the root certificates are different between OS's or it could be because of subtle differences between the SSL implementation of MSFT's schannel. This goes back to one of the reasons we created the extender for in-house use in the first place. Many commercial FTP clients use OpenSSL for SSL protocols but we found that certain FTP servers work better with schannel. The FTPS protocol has enough room for interpretation that there is some variability in which clients work with which servers.

I found that I get the same verification error using your script on both Windows 10 and Windows 11. This adds credence to the first explanation above.

Note that the extender was developed and tested on Windows 11, and tested on Windows 10 and Windows 7, as well.

[td]

I tried using the command line version of openssl on your FTP site and it reported the error "D8070000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:ssl\record\ssl3_record.c:355". This is usually but not always because the site does not support SSL at all.

[edit] The error can also mean the server's TLS protocol is disabled on the client system. This is usually because the server is using an older protocol that Windows considers unsafe and is therefore disabled on the client.

[spl]

Not my site. Both are public FTP used for testing code and checking upload/download speed. And, as I first questioned, could the extender determine a server configuration in advance. Wasn't trying to be the bad guy, just suggesting code for testing.

Maybe Kirby or someone else who has a real FTPs application to support will comment: ITMT: don't dis' sFTP

[td]

We are familiar with the sites and have already used them for testing. Supported TLS protocols are an OS issue since the extender uses the OS's SSL implementation. Most command line tools will just fail when they encounter an older unsupported SSL protocol.

[spl]

Completely understood. [what works]

[td]

My Workstation (Fully updated Windows 11...) supports both TLS 1.2 and TLS 1.3 and I get your ftp.dlptest.com so that TLS protocol can be ruled out as the issue. Also, I can download files from both ftp.dlptest.com and test.rebex.net using the default @fsFTPES protocol. Only fsDirList seems to have the problem. The error for 'test.rebex.net' suggests that the data size reported by the server and the actual the size of the data received by the client are not the same and are considered an error.

An easy way to get an enumeration of supported ciphers and protocols is to use our friend nmap:

nmap --script ssl-enum-ciphers -p 21 ftp.dlptest.com
nmap --script ssl-enum-ciphers -p 21 test.rebex.net

[spl]

Only fsDirList seems to have the problem.

So, fsClientInit(host, port, user, pwd) connects in both cases, and if you know the name of a file to download and can skip fsDirList that will not error. I'm going to play around with either Intcontrol(73,2,0,0,0) or Intcontrol(73,3,0,"geterror",0) in the test scripts I uploaded so as not to throw baby out with bathwater.

[EDIT2]: clumsy, but workable

Code Select Expand

gosub udfs
IntControl(73,1,0,0,0)
try =0
AddExtender('ilcfs44i.dll',0,'ilcfs64i.dll')
host = 'ftp.dlptest.com'
port = 21
user = 'dlpuser'
pwd  = 'rNrKYTX9g7z3RgJRmxWuGHbeu'
client = fsClientInit(host, port, user, pwd)
dir = "/"
listfiles()
:end
fsClientDelete(client)
Exit         

:WBERRORHANDLER
geterror()
Message("Error Encountered",errmsg)
fsClientDelete(client)
Exit         
;/////////////////////////////////////////////////////////////////////////////////////

:udfs
#DefineSubRoutine geterror()
   wberroradditionalinfo = wberrorarray[6]
   lasterr = wberrorarray[0]
   handlerline = wberrorarray[1]
   textstring = wberrorarray[5]
   linenumber = wberrorarray[8]
   errmsg = "Error: ":lasterr:@LF:textstring:@LF:"Line (":linenumber:")":@LF:wberroradditionalinfo
   Return(errmsg)
#EndSubRoutine

#DefineSubRoutine listfiles()
If try== 0
   IntControl(73,2,0,0,0)
Endif
list = fsDirList(client, dir, @fsList)
file = AskItemlist('Select a File', list, @lf, @sorted, @single)
Message('You Selected',file)
fsClientDelete(client)
Exit

:WBERRORHANDLER
If try == 0
   Display(1,"Error Encountered","Retrying")
   fsClientDelete(client)
   client = fsClientInit(host, port, user, pwd, @fsFTP) 
   listfiles()
Else
   try +=1
Endif
Return(0)   
#EndSubRoutine

Return



[td]

Speaking of clumsy another typo. Proofreading has never been a strength. The sentence fragment "I get your ftp.dlptest.com so that TLS protocol can be ruled out" should be "I get your ftp.dlptest.com error so the TLS protocol can be ruled out".

I am not sure that anything can be done to the extender to correct the dsDirList issue with these two sites. It works well with other sites we used in testing, so I suspect it is a protocol interpretation issue on the server side. I will examine the problem in more depth. I might get lucky and find a workaround.

[td]

I got the same error using the curl command line and a similar error when using FileZilla to list files at ftp.dlptest.com. The FileZilla error is interesting because FileZilla uses the GnuTLS library instead of OpenSSl or SChannel. The error in the FileZilla case was "error -15: An unexpected TLS packet was received." Me thinks the problem is on the server side.

{edit} Also tested on one of our Linux boxes using curl which uses OpenSSl on Linux. I got the same error. I also did more reading on the problem and the consensus is that if FTPS gets the error but plain FTP does not, it is a server-side problem.

[spl]

and the consensus is that if FTPS gets the error but plain FTP does not, it is a server-side problem.

yes, and since we all agree, then how best to detect and deal with.

[td]

Haven't you already found a good solution? Your script does appear to do the trick. FWIW, I suspect but do not know with certainty that the problem occurs with one of the common Linux FTP servers. It may be a server configuration problem or a deeper issue with the FTP server code. It does seem to be confined to the FTP NLST command.

[spl]

Sir;

Didn't think this was a contest. I just uploaded to latest WB, you posted a new Extender, I scripted a couple of tests with errors. Stuff like this there is no real solution. Moving on will be submitting code for Power Query and HTML tables, which will probably be ignored.

[td]

I didn't think it was a contest either and I appreciate that you took the time to explore the new extender. You found an interesting issue with some FTPS servers that unfortunately does not have a client-side solution.

I took a peek at your Power Query script and it is interesting as a possible new approach to Web scraping for WinBatch users. I will explore that possibility further when time permits.