Capture Terminal Output

[nrr]

Hello .... I am a volunteer teacher at an "English as a Second Language" course.  with students from different countries in each class, I developed a neat program that uses the DeepL language translation api to perform multi-language translations for any given input.  I use a Windows terminal to submit a curl command to DeepL and pipe the response to the Windows clipboard via the "! clip" option.  I retrieve and parse the clipboard contents for display in my program output.  All this works great except when the translated response contains non-English characters (which is usually the case).   I know I can accomplish my needs by piping the output to a file instead of the clipboard but I'd much prefer a non-file transfer approach.

Does anyone know why the clipboard mangles non-English characters and know if there's some way I can accomplish what I'm trying to do without a file transfer and without any type of send keys approach?

I realize the part of my program that is causing problems isn't a Winbatch problem but I also know from years of past Winbatch experience that this is the best forum on the internet and if there's a solution, someone here will have already done it.

Thanks
Nick

[nrr]

Just to clarify ... the problem occurs with the clipboard.  When the response contains non English characters, the output to the clipboard via the " | clip" option is completely mangled.
Nick

[td]

Based on your description, the output form curl is UTF-8 character encoding which makes sense for multiple reasons. This presents a problem because the Windows clipboard does not natively support UTF-8. This means you would need to figure out a way to convert UTF-8 to UTF16 LE (Windows Unicode) before loading the text onto the clipboard. Curl has more switches than letters in the alphabet. It would be one place to check. You could also consider rethinking your solution. At the risk of being accused of product plugging, WinBatch could likely handle the entire process for you. You could also consider using a Powershell script.

[nrr]

Tony ... I would gladly welcome a Winbatch only solution.  I know how to build the Dialog user interface and how to send the data to Deepl via a Command terminal but then I'm left with the same problem ... how do I get the return data back to my user interface Dialog.  Is there another approach?
Thanks
Nick

[td]

I don't know much about deepl but based on your description the Windows Application is a command line tool. There are several examples of sending input and capturing output from a command application in the Tech Database. WinBatch can also convert UTF-8 to UTF-16 LE. I will need to learn more about using deepl before offering detailed suggestions. I will post back here once I have time to get up to speed.

[nrr]

Thanks for the reply. 
Deepl offers a few programming language interfaces to their API including Python, Java, PHP, .net, and curl.  I first built my solution in Python and it works great.  But I'd rather not have to install Python on any other teacher's computer.  Next I went to Excel/VBA/curl and that works also except for the problem I described with the clipboard.
I'm going to work on a WB solution this weekend.  I don't know .Net but understand from reading the forum and tech database that WB can use .net so I'm going to ask Deepl for sample .net code to send an inquiry.
I appreciate your help and guidance.
Nick

[stanl]

I'm probably off-base here but Power Query in Excel has Internationalization features, and Power Query can be called from WB as I remember a couple of posts with code I made.


But it sounds like you are doing excellent work as a volunteer. I will be looking forward to replies/posts you have and will happily contribute if i can.

[td]

I need to learn more about DeepL before pontificating again but... When curl is mentioned the first thought is the HTTP protocol. If DeepL uses HTTP as a protocol then it should translate to WIL. I have some first-hand experience translating from a curl API to WIL so I know it is in at least specific cases doable.

What is not clear is the role of Excel. Is it just an intermediate step in the process or is it the originating source of the text being translated?

[nrr]

Excel is the source of the request but only because it is an easy way to create a nice user form and because its already available on most teachers' PCs.  A WB-only solution would even be better.
Nick

[nrr]

Tony/Stan ... thank you both for you guidance.  I found a sample .net request to DeepL (pasted below) but I have never used .net and am clueless.  I"m erroring on the very first command ... "using System" is undefined.  Do I need some additional libraries?  (I'm still looking.)
If I can get this working, I'll have a nice WB solution.
Nick
using System;
using System.Net.Http;
using System.Threading.Tasks;

namespace DeepLAPI
{
public class DeepLAPI
    {
        private readonly HttpClient httpClient;
        private readonly string authKey;

      public DeepLAPI(string authKey)
        {
            this.httpClient = new HttpClient();
            this.authKey = authKey;
        }

        public async Task<string> TranslateTextAsync(string text, string sourceLang, string targetLang)
        {
            var url = "https://api.deepl.com/v2/translate";
            var content = new FormUrlEncodedContent(new[]
            {
                new KeyValuePair<string, string>("auth_key", [authKey]),
                new KeyValuePair<string, string>("text", "hello world"),
                new KeyValuePair<string, string>("source_lang", sourceLang),
                new KeyValuePair<string, string>("target_lang", targetLang)
            });
            var response = await httpClient.PostAsync(url, content);
            var responseString = await response.Content.ReadAsStringAsync();
            return responseString;
        }
    }
}

[stanl]

I'm sure Tony, or Jim or Chuck could convert the .net => C# for use with WB. I would also think it might be possible to construct a REST query to https://api.deepl.com/v2/translate with your auth code and additional parameter for the text, source and target languages. Would have to see if DeepL handles REST queries. :-\


EDIT:


https://www.deepl.com/en/docs-api seems instructive. Results returned as Json.

[nrr]

Stan ... much thanks for the Power Query suggestion.  I had never used it previously but it works like a charm.  And with that, I now have a working solution.  The translated text from DeepL is piped to a file instead of the clipboard and the Power Query retrieves the data from the file reliably and quickly.
Thanks to all for the great suggestions.
Nick

[td]

Based on the .Net example, it appears that DeepL provides an HTTP interface using an HTML form. The HTTP/HTML is relatively straightforward and could be duplicated in WIL without CLR hosting (.Net).

[nrr]

So I built a quick WB script to POST a request to DeepL but I'm getting a 403 error response and I see this on their site ...

CORS Requests
If you try to send requests to the DeepL API from the browser, your requests will fail with the HTTP 403 Forbidden status code with a message stating "blocked by CORS policy". The DeepL API does not allow calls directly from browser-based applications.

Yet I can follow their examples an easily post a GET request directly from a browser. 

I've sent them an email requesting clarification.
Nik

[td]

That is a bit unusual. I wouldn't know what to make of it either.

It is likely of little use but here is a completely untested quick and dirty script that attempts to approximate one of the DeepL examples.  It needs work and is almost guaranteed not to work without a few corrections.

Code (winbatch) Select Expand

objHttp=ObjectCreate("WinHttp.WinHttpRequest.5.1")

URL = "https://api-free.deepl.com/v2/translate HTTP/2"
objHttp.SetTimeouts(50000, 50000, 50000, 50000);
objHttp.open("POST",URL,"False")
objHttp.SetRequestHeader("Authorization", "DeepL-Auth-Key [yourAuthKey]" ) ; Put your key here!
objHttp.SetRequestHeader("User-Agent", "YourApp/1.2.3") ; Put your "app" name here!
objHttp.SetRequestHeader("Content-Length", "71")
objHttp.SetRequestHeader("Accept", "application/json")
objHttp.SetRequestHeader("Content-Type", "application/json")
Body = `{"text":["The table is green. The chair is black."],"target_lang":"DE"}`
objHttp.Send(body)

Terminate(objHttp.Status != 200,"DeepL Translator", "Translation request failed")

strResponse=objHttp.responseText
objHttp=0

; The use of the JSON extender is not necessary but it makes
; text extraction simpler.
AddExtender("ilcjs44i.dll",0,"ilcjs64i.dll")       

hResponse=JsParse(strResponse)
strTranslated = jSValueGet(hResponse,"translations[0].text")

exit


[nrr]

Tony ... I had taken the WWINET route and my code is similar to yours.  You have some things as headers that I had a urlData.
I input my auth key and changed the url to point to the production version.  I'm getting a 404 error but will work more on this tomorrow.

While I do have the Excel solution working, I found to day that it didn't work on my older PC.  That PC had a version of Excel that pre-dated the Power Query.  So I'm still hoping for the WB solution.
thanks.
Nick

[stanl]

While I do have the Excel solution working, I found to day that it didn't work on my older PC.  That PC had a version of Excel that pre-dated the Power Query.  So I'm still hoping for the WB solution.
thanks.
Nick

Not sure how old the other PC version is, but Office 2013 allows you to download power query as an add-on.

[nrr]

Thanks Stan. MS download site says it is now only supported in Office 2016 or later.  But it is still helpful to know that.  My guess is that most of the teachers will have 2016 or later.
Nick 

[td]

It goes without saying that it is better to avoid as many external, non-OS dependencies as possible in scripts you intend to use long term or provide to others. However, it is sometimes unavoidable. It would be great to know if or when you crack the direct HTTP API nut.

[nrr]

So far I only get back a 404 error code no matter what I try for the host or api key.  I will wait for DeepL support to get back to me and will definitely post a response whe I hear from them.
Nick

[stanl]

Thanks Stan. MS download site says it is now only supported in Office 2016 or later.  But it is still helpful to know that.  My guess is that most of the teachers will have 2016 or later.
Nick

Ok. I was about to say Power Query can also be downloaded as add-on for Office 2010. But for 2010/2013 it is configured as a separate bar in Excel. But to Tony's point. I would assume you would have to build an exe for other 'users' and those have their own problems, given components user has.


But this does appear to be moving forward.

[stanl]

So far I only get back a 404 error code no matter what I try for the host or api key. 
Nick

Another stab: could the protocol have anything to do with the 404 error. Possibly they need Tls12 which WB can handle.

[td]

At least the error has changed from 403 to 404. The 404 error does suggest there is at least marginal progress. 

[nrr]

Still no response from DeepL support but in the mean time I've developed a fully WB-only solution.  I encountered some terminal window weirdness and have to use wt.exe instead of cmd.exe but other than that, the app works just fine.

I direct the DeepL output to a file and I can read it thanks to this code gem (nPrevCP = ChrSetCodepage(65001)) that I found on another forum post. I parse the translated text and display it in my UI in a ReportView control.  Looks pretty cool as the responses populate the RV control one-by-one (but very quickly).
Nick

[td]

I should have mentioned ChrSetCodepage(65001) in an earlier post. It would have saved you searching for it.

I am surprised you need to use wt.exe instead of cmd.exe. When I run cmd.exe on my system, I get a Windows terminal window (wt.exe) with a cmd.exe tab. But that could be because of how I have wt.exe configured or how you are starting the cmd/wt process.

[nrr]

RE: wt/exe ... I was having weird problems trying to submit my curl command to DeepL using various Run, ShellExecute, or wScript.Run commands.  My app would sometimes freeze and I'd have to reboot.  Other times the terminal window would scroll rapidly with old curl commands.   I'd have to Ctrl-C to make it stop.

Finally I found that wScript.Exec would work but there is not option to run hidden.    The terminal window flashed annoyingly with each submitted query.

To further test, i created a new WB file and copied these two lines of code from my app into my new wb app ...

strCommand = 'curl -X POST "https://api.deepl.com/v2/translate" --header "Authorization: DeepL-Auth-Key 1edca10d-4069-d1fe-76fc-2687a1287160" --data-urlencode "text=what is your name" --data-urlencode "target_lang=FR" -o "C:\ESL\Translator.txt"'   
ShellExecute("cmd.exe",strCommand,"", @Normal,"")

This causes my app to freeze but it runs just fine when I run it from my new simple test app, it executes correctly.   In other words ... two .wbt files containing the exact same code, when run from the studio, one works and the other one fails.

I found a reference to the wt.exe from Chuck in a recent other post.  I changed cmd.exe to wt.exe and can run it @Hidden in my app.  This give me just the appearance that I wanted.  I just wonder what will work on other teachers' PCs.

Thanks for all your help.
Nick
Yes, I left my DeepL auth key in the query.  Don't tell anyone.

[td]

Perhaps it's a random quantum event multiple universes problem? I took the liberty of running your posted lines in WinBatch Studio multiple times without any issues. Out of habit, I did add the command line switch "/C" before the "curl". I did try the line with the @Hidden option and it worked, as well. Tried with a compiled version and that worked too.

There may be something else in your script that causing it to hang but you seem to have a workaround in place.

[td]

Just a random thought. Could you have a synchronous/asynchronous problem of some kind? If so, perhaps try ShellExecuteWait.

[stanl]

I ran this through Powershell

Code Select Expand



Clear-Host
Add-Type -AssemblyName System.Web
$APIKey              ="your API Key"
$textToConvert       = "Sir, where is the Library?"
$translateBaseURI    = "https://api.deepl.com/v2/translate"
$TargetLang          = "fr"
$headers             = @{Authorization = "DeepL-Auth-Key $APIKey"}
$Body                = @{target_lang = $TargetLang}
$Body               += @{source_lang = "EN"}
$Body               += @{split_sentences = 0}
$Body               += @{text = $textToConvert}
$Response            = Invoke-RestMethod -Uri $translateBaseURI -Method Post -Body $Body -Headers $headers
$Translation         = [System.Web.HttpUtility]::HtmlDecode([Text.Encoding]::UTF8.GetString([Text.Encoding]::GetEncoding(28591).GetBytes($Response.translations.text)))
$Translation



and got



Monsieur, où se trouve la bibliothèque ?

[td]

I got the above COM Automation script using the "WinHttp.WinHttpRequest.5.1" object working as well. The only changes were to the headers Authentication DeepL-Auth-Key and User-Agent.

[stanl]

Not a competition [AFAIK]. Just wanted to illustrate that the basic REST query is valid.

[nrr]

Stan and Tony ... YOU ARE THE KING(S)!.  I can't thank you enough for your guidance and patience.  I though I'd tried every possible combination of URL, headers, and body to get this working but I just always got a 403 or 404 error until today.

Since you've been so interested in my app's progress, if you'll allow, here's a bit about my long WB journey ...

I became interested in WB way back in the mid-90's when I saw what a colleague was able to accomplish with WB.  While my "real" job was as a java developer, I continued my pursuit of learning WB over the years.  Long story short, in 2008, my employer offered me a voluntary severance package.  While I contemplated accepting the package, a recruiter contacted me about a client that desparately needed a WB programmer.  Apparently the Revenue Cycle Manger at his client, Stanford University Medical Center, who had developed a complete system of WB programs to handle their mission-critical revenue cycle process had resigned.  To make matters worse, the first person Stanford hired had misrepresented his WB skill and was fired.  How lucky was that for me??
 
I accepted the package from my employer and started the Stanford gig not knowing if it would last a few weeks or months. For the next 8 years until I retired, I enjoyed the best job I ever had supporting and enhancing the Stanford programs to completely automate handling claim and remittance files.  They thought I was a hero and actually I was just a lucky programmer who happened to learn WB.

Anyway, thanks again.  I'll be demonstrating this app to my fellow teachers at our local English Second Language program early in January.

If you'd like to share your WB stories, I'd love to hear them.

Happy Holidays,
Nick

 

[stanl]

Too old to tell stories. But if your app can be compiled and shared with others that is a plus. One of the benefits of WB compiled :)


[EDIT]


but, I will say this has been a great thread.... I edited the Powershell to ask
"Hey, who ate my Lobster"? translated to German [de]


made my day

[td]

Not a competition [AFAIK]. Just wanted to illustrate that the basic REST query is valid.

I didn't think it was. I was attempting to add another option to the mix. I could have changed the COM example to URL data. But being lazy and since using headers worked, I didn't bother.

[nrr]

I still can't get the WinHTTP version to properly display the translated text.  It woks great when using WT to Powershell but response from WinHttp is garbled.
I've tried everything about code pages and UTF versions so if you have a hint, I'd appreciate it.
Thanks
Nick