Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Stop the IdHTTP.Post stream upload without closing the connection
#1
Hi

Let's see if I can phrase my question well enough.
I use TIdHttp.Post (and Patch) requests to upload a source stream, like

Code:
fHTTP.Patch(URL, InputStream);

Is there some way to interrupt the uploading of the stream, as if the data has ended, but without raising exceptions and closing the connections?

I am migrating some Java code that deals with a tus server. And one of the methods seems to do exactly the above. It make a patch request, uploads a stream, but it also has the option to stop the upload. When it is stopped, the client stops uploading the stream, but afterwards still reads the response from the server over the same connection. That response contains information about the data uploaded.

Looking at the code I could not find a clever solution. The uploading of the stream is called by TIdCustomHTTP.ConnectToHost and handled inside TIdIOHandler.Write and WriteDirect. But I didn't see how it can be interrupted midway without raising an exception, which then closes the connection.

Using Delphi 10.1 with Indy that comes with it.

Thanks.
Reply
#2
(01-24-2019, 07:16 PM)vladimir Wrote: Is there some way to interrupt the uploading of the stream, as if the data has ended, but without raising exceptions and closing the connections?

No, there is not.

(01-24-2019, 07:16 PM)vladimir Wrote: I am migrating some Java code that deals with a tus server. And one of the methods seems to do exactly the above. It make a patch request, uploads a stream, but it also has the option to stop the upload. When it is stopped, the client stops uploading the stream, but afterwards still reads the response from the server over the same connection. That response contains information about the data uploaded.

In standard HTTP, an HTTP server cannot send a response until the request is finished (unless an error occurs, then a response can be sent sooner). The only way an HTTP client can stop a request prematurely without breaking the connection is if the client is sending an HTTP 1.1 request that contains body data encoded in the 'chunked' transfer encoding, and simply stops sending chunks, sending a 0-length chunk to end the body. HTTP 1.1 allows this for requests, but TIdHTTP does not currently support sending chunked requests, only reading chunked responses.

However, tus is not using standard HTTP as-is. It defines its own protocol on top of HTTP to handle partial and resumable uploads beyond what HTTP provides (by utilizing a custom application/offset+octet-stream media format for the Content-Type header, and custom Upload-Offset and Upload-Length headers). It does not utilize HTTP's 'chunked' feature.

The actual tus protocol is defined on the tus website:

resumable upload protocol

Based on what I see in that protocol spec, the only provision I see for stopping an upload in progress is the following:

Quote:The tus protocol is built upon the principles of simple pausing and resuming. In order to pause an upload you are allowed to end the current open request. The Server will store the uploaded data as long as no violations against other constraints (e.g. checksums) or internal errors occur.

But, it does not say HOW to "end the current open request" exactly. The only way HTTP allows ending a request prematurely is to close the connection, unless 'chunked' is being used for the body data.

Otherwise, the tus protocol does allow an upload to be broken up into smaller pieces, each one sent with its own PATCH request:

Quote:The Client SHOULD send all the remaining bytes of an upload in a single PATCH request, but MAY also use multiple small requests successively for scenarios where this is desirable. One example for these situations is when the Checksum extension is used.

So, maybe your Java client is simply doing that, and ends the "current" PATCH request normally when requested to stop uploading.

Another possibility is for a tus client to simply stop sending without actually ending the request, letting the server timeout on its end.

Quote:Both, Client and Server, SHOULD attempt to detect and handle network errors predictably. They MAY do so by checking for read/write socket errors, as well as setting read/write timeouts. A timeout SHOULD be handled by closing the underlying connection.

A tus server is required to store whatever data it receives, even if errors occur. If the client intentially causes a timeout, the connection is still open, and maybe the server sends a (failed) response before closing the connection on its end.

Who knows what your Java client is actually doing. You will just have to study its source code, or sniff its HTTP traffic, and see for yourself.

(01-24-2019, 07:16 PM)vladimir Wrote: Looking at the code I could not find a clever solution. The uploading of the stream is called by TIdCustomHTTP.ConnectToHost and handled inside TIdIOHandler.Write and WriteDirect. But I didn't see how it can be interrupted midway without raising an exception, which then closes the connection.

Correct. There is currently no supported option to interrupt an upload in progress without breaking the connection. If TIdHTTP does not suit your needs, you can always use TIdTCPClient instead, and then you have full control to implement the HTTP and tus protocols however you want.

Reply
#3
Thanks for the confirmation.
I expected that it was not possible with the current implementation. I also considered writing a custom client based on TCP, but it was not worth it for the current usage. Maybe I will make a custom IOHandler when the need comes.
Reply
#4
Remy
If I decide to implement the chunked uploading mode, can you give me some hints on how to approach that? E.g. what classes can I inherit and reuse, if possible, and what I would have to rewrite.
Thanks.
Reply
#5
(01-26-2019, 11:04 AM)vladimir Wrote: If I decide to implement the chunked uploading mode

I don't know if tus allows HTTP-style chunked uploads. Nothing in the tus protocol spec describes that.

Unless you are referring to sending your own "chunks" using separate PATCH requests? Which is documented.

(01-26-2019, 11:04 AM)vladimir Wrote: can you give me some hints on how to approach that? E.g. what classes can I inherit and reuse, if possible, and what I would have to rewrite.

You don't need to derive or rewrite anything. The standard TCP/IOHandler components will suffice. What is important is the data you pass in for each request you send. You are responsible for choosing/formatting that data regardless of how you eventually send it.

Reply
#6
(01-27-2019, 02:32 AM)rlebeau Wrote: I don't know if tus allows HTTP-style chunked uploads. Nothing in the tus protocol spec describes that.
Unless you are referring to sending your own "chunks" using separate PATCH requests?  Which is documented.
Yes, after reading some more, what they mean by chunk upload is not the HTTP chunked transfer mode. Just a series of independent patch requests, reusing the connection. So just a misunderstanding on my part.

May I use this thread to ask you something else.

I use an SSL IOhandler. The main Post and Patch calls are executed in a thread. When I want to interrupt them and terminate the thread I called HTTP.Disconnect from the main thread, which causes AVs almost every time. The call stack shows that probably some SSL object was being freed by the disconnect, while still being used.
A search showed me that this is an old problem, e.g. https://stackoverflow.com/questions/5278...n-aborting


In the above SO answer you recommend using Socket.Binding.CloseSocket, instead of Disconnect. Is that still the recommended workaround, considering that the connection might be in a different state (e.g. trying to connect, or transferring)?
Something like

Code:
  if Assigned(fHTTP.Socket.Binding) then
    fHTTP.Socket.Binding.CloseSocket;

This works, but raises EIdSocketError 10053, and I'm not sure whether its safe to ignore those.
Or maybe EIdNotASocket, if I close it before it has connected.
Reply
#7
And come to think of it, what is the best way to send a Post/Patch with a stream, not from the beginning of the stream, but from the current stream position.
The IOHandler Write() method has parameters to control that, but the way it is called from TIdHTTP it always resets the stream pos to 0.
Is there any method or property that I have missed that allows control of it from TIdHTTP?

Thanks.
Reply
#8
(01-27-2019, 12:07 PM)vladimir Wrote: I use an SSL IOhandler. The main Post and Patch calls are executed in a thread. When I want to interrupt them and terminate the thread I called HTTP.Disconnect from the main thread, which causes AVs almost every time. The call stack shows that probably some SSL object was being freed by the disconnect, while still being used.

Yes, it is a known issue when using an SSLIOHandler, as you discovered.

(01-27-2019, 12:07 PM)vladimir Wrote: In the above SO answer you recommend using Socket.Binding.CloseSocket, instead of Disconnect. Is that still the recommended workaround, considering that the connection might be in a different state (e.g. trying to connect, or transferring)?

Yes. But you are responsible for keeping track of the state.

(01-27-2019, 12:07 PM)vladimir Wrote: This works, but raises EIdSocketError 10053, and I'm not sure whether its safe to ignore those.

Yes. That is just a standard "Connection aborted" error.

(01-27-2019, 12:07 PM)vladimir Wrote: Or maybe EIdNotASocket, if I close it before it has connected.

Closing the connection while it is still connecting can be a bit tricky, as that definitely suffers from race conditions. The socket may not have been allocated yet, or it may not have started connecting yet. Probably best to keep track of the state and not even try to close it until after it has fully connected first, then close it.

(01-28-2019, 09:11 PM)vladimir Wrote: And come to think of it, what is the best way to send a Post/Patch with a stream, not from the beginning of the stream, but from the current stream position.

The IOHandler Write() method has parameters to control that, but the way it is called from TIdHTTP it always resets the stream pos to 0.

Is there any method or property that I have missed that allows control of it from TIdHTTP?

No, you haven't missed anything. The input stream is sent in full each time. If you don't want to make copies of your data in separate streams, you will have to wrap the main stream inside of helper streams that simply expose access to subsets of the main stream's data. There are third party stream classes for that purpose, or you can write your own (not that hard to do). In fact, you might be able to utilize Indy's own TIdHTTPRangeStream class for this, though this is not its primary function (it is meant to be used with TIdHTTPServer instead).

Reply
#9
Thanks for the info. I will work on it.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)