Atozed Forums

Full Version: How to send POST data to a webserver?
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Pages: 1 2
This is a continuation of a thread at Embarcadero forum, which no longer responds to my input...


Background:
I have programmed an embedded IoT device (ESP8266) for which I have created a configuration application using FPC/Lazarus with Indylaz (Indy 10).
All of this works fine after some help from Remy L.
But there is one thing I also want to incorporate in the config application and this is firmware update via the network of the IoT device.
The ESP8266 has libraries with a lot of functions including web firmware updating, which I have included in my system.
It works as a simple web server operating on the http port of the IoT device, one simply opens the URI http://<ip address of IoT>/firmware and a minimal form is presented where there are two buttons, one file select button to select which firmware file to use and an Update button which starts the firmware update process.

The form presented by the IoT device on the URI shown above has the following html source:
Code:
<html><head><title>The ESP8266 web updater</title></head>
     <body><b>WiFi module firmware updater</b>
     <br>Select firmware file, then click the Update button!<br>
     <form method='POST' action='' enctype='multipart/form-data'>
                  <input type='file' name='update'>
                  <input type='submit' value='Update'>
               </form>
         </body></html>

The code I have used in Lazarus to try and send the data as a web browser would do looks like this after a brief discussion with Remy:
Code:
function TConfigCommHandler.UploadFirmware(FileName: string; URL: string): boolean;
var
  HTTP: TIdHTTP;
  Src: TIdMultipartFormDataStream;
begin
  Result := false;
  if not FileExists(FileName) then exit;
  HTTP := TIdHTTP.Create;
  try
    Src := TIdMultipartFormDataStream.Create;
    try
      Src.AddFile('update', FileName);
      try
        HTTP.Request.Username := FUserName;
        HTTP.Request.Password := FPassword;
        HTTP.Post(URL, Src);
        Result := true;
      except
        on E: Exception do
          FLastError := 'Exception: ' + E.Message;
      end;
    finally
      Src.Free;;
    end;
  finally
    HTTP.Free;
  end;
end;

Behavior:
When I use the form in Firefox and select the file then hit Update, the result is directly shown on the IoT device's debug serial port by a message that it has started the update. Then when the file data has been fully received and verified the IoT device resets and the new firmware starts running. This works fine but involves a web browser...

When I use the method above then there is no error message or such generated, and there is also no debug message from the IoT device saying that update is in progress. In fact once the method is done nothing has happened on the IoT device. It looks like the IoT server is quietly disregarding the call.

So something is clearly missing here, but what?
How does one simulate the action of a form submission in a web browser but using Indy components in a FPC/Lazarus program?
(04-30-2018, 09:56 PM)BosseB Wrote: [ -> ]This is a continuation of a thread at Embarcadero forum, which no longer responds to my input...

That is because the forums server has been offline for a few days is no longer usable.

(04-30-2018, 09:56 PM)BosseB Wrote: [ -> ]When I use the method above then there is no error message or such generated, and there is also no debug message from the IoT device saying that update is in progress. In fact once the method is done nothing has happened on the IoT device. It looks like the IoT server is quietly disregarding the call.

So something is clearly missing here, but what?

At first glance, the code you have shown looks OK, provided you are posting to the same URL that generates the HTML webform, since that is where the <form> element is asking web browsers to post to.

I suggest you run a packet sniffer (such as Wireshark) on your client machine and look at the actual HTTP traffic that a web browser generates vs what TIdHTTP generates when communicating with your IoT device.  There is obviously going to be a difference between the two, but you are not going to see that just looking at code.

One difference that comes to mind would be if the IoT device uses cookies during the webform submission.  Your code is not currently handling that possibility.  Simply set TIdHTTP.AllowCookies=True, then call TIdHTTP.Get() to retrieve the HTML webform and any associated cookies, before then calling TIdHTTP.Post() to post the webform.  It will automatically send back any matching cookies.

Another possible difference is some web servers don't like Indy's default "User-Agent" request header.  If the IoT device is similar, you might need to change the value of the TIdHTTP.Request.UserAgent property to mimic a known web browser.

(04-30-2018, 09:56 PM)BosseB Wrote: [ -> ]How does one simulate the action of a form submission in a web browser but using Indy components in a FPC/Lazarus program?

You already are.  The correct solution is to use TIdHTTP.Post() with a TStrings or TIdMultipartFormDataStream, depending on the "enctype" of the <form> element.  You just have to make sure that you are including everything in your request that a web browser would include.
Thanks, I have modified the code as follows and tested it:
Code:
function TConfigCommHandler.UploadFirmware(FileName: string; URL: string): boolean;
var
 HTTP: TIdHTTP;
 Src: TIdMultipartFormDataStream;
begin
 Result := false;
 if not FileExists(FileName) then exit;
 HTTP := TIdHTTP.Create;
 try
   Src := TIdMultipartFormDataStream.Create;
   try
     Src.AddFile('update', FileName);
     try
       HTTP.Request.Username := FUserName;
       HTTP.Request.Password := FPassword;
       HTTP.AllowCookies := true; // * Added to try make the server accept update
       HTTP.Get(URL);  // * Added to read back form to simulate the action of a web browser
       HTTP.Post(URL, Src);  //Send the update file
       HTTP.Get(URL);  // * Added to retrieve the response
       Result := true;
     except
       on E: Exception do
         FLastError := 'Exception: ' + E.Message;
     end;
   finally
     Src.Free;;
   end;
 finally
   HTTP.Free;
 end;
end;
Still no update but I got a bit further, now the debug serial output shows a message "sleep disable" when the transfer starts and this is like it does when run from the browser too.

Quote:I suggest you run a packet sniffer (such as Wireshark) on your client machine and look at the actual HTTP traffic that a web browser generates vs what TIdHTTP generates when communicating with your IoT device. There is obviously going to be a difference between the two, but you are not going to see that just looking at code.
I have not used Wireshark to do any analysis. Many years ago I installed it but I did not really understand how to use it. There simply was too much stuff being shown. Is there a way to set it up to ONLY show traffic to and from a specific IP address?
I will start by downloading a more recent version in any case.

PS: I do have the C++ source code for the WebUpdater in the IoT device, only I am not a C++ coder so I cannot make much out of it. DS
(05-01-2018, 06:30 AM)BosseB Wrote: [ -> ]Thanks, I have modified the code as follows and tested it:

The 2nd call to TIdHTTP.Get() is unnecessary.

(05-01-2018, 06:30 AM)BosseB Wrote: [ -> ]I have not used Wireshark to do any analysis. Many years ago I installed it but I did not really understand how to use it. There simply was too much stuff being shown. Is there a way to set it up to ONLY show traffic to and from a specific IP address?

Of course. Wireshark has a fairly powerful filtering engine. Plus, once you see any packet for a given TCP connection, you can simply right-click on it and choose "Follow TCP stream", and that will filter just the packets for that specific connection, and will also open a separate window with just the application-level data (HTTP in this case) with all the lower-level data hidden.

You don't have to use Wireshark at all, that was just one example. You could alternatively use your web browser's built-in debugger (most modern browsers have one, including Firefox) to see the browser's HTTP traffic, and assign one of Indy's TIdLog... components to the TIdHTTP.Intercept property to see Indy's HTTP traffic. Then you can compare the two outputs for differences and adjust your TIdHTTP code accordingly.
I have switched on debugging in FireFox and run a session via the web form.

This is what I caught:
HEADERS:
Request headers:

Accept text/html,application/xhtml+xm…plication/xml;q=0.9,*/*;q=0.8
Accept-Encoding gzip, deflate
Accept-Language en-US,en;q=0.5
Authorization Basic YWdpYWRtaW46YWdpdXNh
Connection keep-alive
Content-Length 304673  (Actual file length in bytes is: 304464 bytes)
Content-Type multipart/form-data; boundary=…----------------2651643628205
Host 192.168.119.249
Referer http://192.168.119.249/firmware
Upgrade-Insecure-Requests 1
User-Agent Mozilla/5.0 (Windows NT 6.1; W…) Gecko/20100101 Firefox/59.0

Response headers
Connection close
Content-Length 75
Content-Type text/html

PARAMS:
-----------------------------2651643628205
Content-Disposition: form-data; name="update"; filename="DHT_MONITOR.bin"
Content-Type: application/octet-stream
( then follows what appears to be the content of the bin file )
-----------------------------2651643628205--

Response:
Update Success! Rebooting...

Response payload:
<META http-equiv="refresh" content="15;URL=/firmware">Update Success! Rebooting...

Is there something here that I should add into the TIdHTTP component before using POST?
Like this:
Code:
        HTTP.Request.Accept := 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8';
        HTTP.Request.AcceptEncoding := 'gzip, deflate';
        HTTP.Request.AcceptLanguage := 'en-US,en;q=0.5';
        HTTP.Request.ContentType := 'multipart/form-data;';
        HTTP.Request.UserAgent := 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0';


I have noted that the Content-Length is always bigger than the actual file size and the difference seems to be close to the same but not exactly so...

I was unable to use Wireshark, could not even find how to set a filter for the IP address or start capturing...
(05-01-2018, 07:43 PM)BosseB Wrote: [ -> ]Is there something here that I should add into the TIdHTTP component before using POST?

You should add the "Referer" request header. You can use the TIdHTTP.Request.Referer property for that:

Code:
HTTP.Request.Referer := 'http://192.168.119.249/firmware';

That header tells the server which URL is submitting the webform to the server. Some servers do perform Referer validation to make sure only valid Referers can send in data.

(05-01-2018, 07:43 PM)BosseB Wrote: [ -> ]
Code:
HTTP.Request.Accept := 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8';

You don't really need that header. All it is doing is telling the server that the browser prefers (X)HTML and XML, but will still accept anything. Indy's default Request.Accept property value will suffice.

(05-01-2018, 07:43 PM)BosseB Wrote: [ -> ]
Code:
HTTP.Request.AcceptEncoding := 'gzip, deflate';

That header, you definitely SHOULD NOT set manually. It is telling the server that the browser will accept compressed responses. TIdHTTP handles compression but ONLY when its Compressor property is configured. Let TIdHTTP manage its Request.AcceptEncoding property for you, based on TIdHTTP's actual capabilities.

(05-01-2018, 07:43 PM)BosseB Wrote: [ -> ]
Code:
HTTP.Request.AcceptLanguage := 'en-US,en;q=0.5';

You don't need that.

(05-01-2018, 07:43 PM)BosseB Wrote: [ -> ]
Code:
HTTP.Request.ContentType := 'multipart/form-data;';

You don't need to set the Request.ContentType property manually when calling Post(). It will assign an appropriate ContentType for you, depending on the type of data being posted.

(05-01-2018, 07:43 PM)BosseB Wrote: [ -> ]
Code:
HTTP.Request.UserAgent := 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0';

That is optional, depending on if/how the server reacts to different User-Agent strings.

(05-01-2018, 07:43 PM)BosseB Wrote: [ -> ]I have noted that the Content-Length is always bigger than the actual file size and the difference seems to be close to the same but not exactly so...

The "Content-Length" reflects the total size of the entire data that is being posted. It includes the MIME data that surrounds your file data. You are not posting just the file by itself, there is other data being posted, too (ie, the file's name and media type).

(05-01-2018, 07:43 PM)BosseB Wrote: [ -> ]I was unable to use Wireshark, could not even find how to set a filter for the IP address or start capturing...

The filter is right there at the top of the main window. It says "Filter:" next to it.

You use the "Capture" menu to choose which network interface(s) to capture from and with what settings, and to start/stop captures.

I suggest you read Wireshark's documentation, particularly the User's Guide.
Thanks Remy,
I added the Referer and commented out the other added items, then tested.
Unfortunately there is something else that causes the update not fail without error/exception.
The IoT device just ticks along without regard to the firmware being sent over.
Late now so I will continue tomorrow.
(05-01-2018, 07:43 PM)BosseB Wrote: [ -> ]I have switched on debugging in FireFox and run a session via the web form.

This is what I caught:

And, have you captured TIdHTTP's output, like I described earlier?

Quote:assign one of Indy's TIdLog... components to the TIdHTTP.Intercept property to see Indy's HTTP traffic

It is not worth capturing the web browser's output if you don't also capture TIdHTTP's output for comparison.

Based on the code you have shown so far, and the changes I suggested, TIdHTTP should be outputting something like this:

Code:
POST /firmware HTTP/1.0
Connection: keep-alive
Content-Type: multipart/form-data; boundary=--------050118161753994
Content-Length: 304674
Host: 192.168.119.249
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Referer: http://192.168.119.249/firmware
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0

----------050118161753994
Content-Disposition: form-data; name="update"; filename="DHT_MONITOR.bin"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary

<file data here>
----------050118161753994--

One thing I notice that is missing from TIdHTTP's output that is present in Firefox's output is the "Authorization" header. By default, TIdHTTP will not transmit authentication credentials unless the server asks for them (in which case, you need to enable the hoInProcessAuth flag in the TIdHTTP.HTTPOptions property), or you set the TIdHTTP.Request.BasicAuthentication property to True (it is False by default).

So, try enabling authentication in your TIdHTTP so the Request.Username and Request.Password are actually sent to the server. TIdHTTP's output should then look more like this:

Code:
POST /firmware HTTP/1.0
Connection: keep-alive
Content-Type: multipart/form-data; boundary=--------050118161840615
Content-Length: 304674
Host: 192.168.119.249
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Referer: http://192.168.119.249/firmware
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0
Authorization: Basic YWdpYWRtaW46YWdpdXNh

----------050118161840615
Content-Disposition: form-data; name="update"; filename="DHT_MONITOR.bin"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary

<file data here>
----------050118161840615--
(05-01-2018, 10:35 PM)rlebeau Wrote: [ -> ]And, have you captured TIdHTTP's output, like I described earlier?
assign one of Indy's TIdLog... components to the TIdHTTP.Intercept property to see Indy's HTTP traffic
Not yet, I have never used the log component before and now when I look at the Indy10 docs I see that there are several such object to choose from. Don't know how to set it up and how to view its output...

Quote:One thing I notice that is missing from TIdHTTP's output that is present in Firefox's output is the 'Authorization' header.  By default, TIdHTTP will not transmit authentication credentials unless the server asks for them (in which case, you need to enable the hoInProcessAuth flag in the TIdHTTP.HTTPOptions property), or you set the TIdHTTP.Request.BasicAuthentication property to True (it is False by default).

So, try enabling authentication in your TIdHTTP so the Request.Username and Request.Password are actually sent to the server.  TIdHTTP's output should then look more like this:
My mistake, I thought that setting the user and password for IdHTTP would enable the authentication, so I did nothing else. Now I have enabled BasicAuthentication and tested again, but it seemingly made no difference.

Wireshark
I managed to set up a filter to capture the traffic on the http port with the IoT device IP address so I could capture the  actual data both for the FireFox case and using my FPC config application:

Code:
tcp port http and host 192.168.119.249

But these captures could only be viewed inside of Wireshark and then only one at a time, so it is very difficult to compare for differences. Tried to load a saved log into a text editor but that only showed unreadable data even though I saved to a text file format...


But I can see a definite difference towards the end of the logs, but I don't know what is relevant and not.
(05-02-2018, 12:45 PM)BosseB Wrote: [ -> ]I have never used the log component before and now when I look at the Indy10 docs I see that there are several such object to choose from. Don't know how to set it up and how to view its output...

The easiest one to use would be TIdLogFile. Simply set its FileName property to the full path of an output file of your choosing, and its Active property to True. Then assign it to the TIdHTTP.Intercept property. Everything the TIdHTTP reads from, and writes to, the socket will be written to the log file. Then you can view the file in whatever text editor you want.

(05-02-2018, 12:45 PM)BosseB Wrote: [ -> ]I managed to set up a filter to capture the traffic on the http port with the IoT device IP address so I could capture the  actual data both for the FireFox case and using my FPC config application:

Code:
tcp port http and host 192.168.119.249

A simpler filter would be:

Code:
http && host==192.168.119.249

(05-02-2018, 12:45 PM)BosseB Wrote: [ -> ]But these captures could only be viewed inside of Wireshark and then only one at a time, so it is very difficult to compare for differences.

You can run multiple instances of Wireshark at a time. Save each capture to a separate .pcap file, and then open each file in a separate Wireshark instance so you can view them side-by-side.

Or, once you have the captures, you can export (not save) just the captured HTTP data to separate .txt files, and then you can open them in whatever text viewer you want.

(05-02-2018, 12:45 PM)BosseB Wrote: [ -> ]Tried to load a saved log into a text editor but that only showed unreadable data even though I saved to a text file format...

Then you didn't actually save it to a text format.
Pages: 1 2