Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
IndyTextEncoding_8Bit Issue?
#1
Greetings,

I am trying to convert the TCP classes in an existing Delphi 5 application to Indy so that I can add SSL. The application sends text as well as binary commands to a server as strings over a TCP connection. The commands have a wide range of values outside of the normal ASCII character range (0x80 to 0xFF). I have configured the IOHandler to use IndyTextEncoding_8Bit encoding, however it appears that values from 0x80 to 0x9F are still being sent as 0x3F (the '?' which I believe is sent when using sending non-ASCII chars with ASCII encoding). I have created a simple test client/server app to recreate the behavior that I'm seeing. Here is the relevant client code:

Code:
constructor TestClient.Create();
begin
  inherited Create;
  TCPC := TIdTCPClient.Create();
  TCPC.ConnectTimeout := 5000;
  TCPC.Host := 'localhost';
  TCPC.Port := 5000;
  try
    TCPC.Connect;
    TCPC.IOHandler.DefStringEncoding := IndyTextEncoding_8Bit;
  except
    on Exception do begin
      ShowMessage('error occurred during connect');
    end;
  end;
end;

procedure TForm1.Button2Click(Sender: TObject);
  var s: string;
begin
  s := #128;
  RunClient.TCPC.IOHandler.Write(s);
end;

The server has also been configured to use IndyTextEncoding_8Bit encoding and is receiving the expected hex values when sending data outside of the 0x80 to 0x9F range. I have also verified with Wireshark that that 0x3F is being sent by the client when running the above code.

Is there a different encoding type that I should be using? or some issue with the 0x80 to 0x9F value range? perhaps an incompatibility when using Indy with Delphi 5? I've been pulling my hair out and I'm hoping that I'm missing something obvious (I'm new to Delphi so that could easily be the case!).

-Ed
Reply
#2
(01-09-2023, 03:49 AM)eddie1473 Wrote: The commands have a wide range of values outside of the normal ASCII character range (0x80 to 0xFF). I have configured the IOHandler to use IndyTextEncoding_8Bit encoding

Why are you using the 8bit encoding at all, instead of, say, UTF-8 instead?  The 8bit encoding is not intended for the way you are using it.  Communication protocols should use standardized encodings, and UTF-8 is well-suited for that purpose.

In any case, in pre-Unicode versions (which includes Delphi 5), where String is an alias for AnsiString, you need to utilize the IOHandler's DefAnsiEncoding property as well as its DefStringEncoding property.  The DefStringEncoding property is used only to convert between bytes and Unicode strings, whereas the DefAnsiEncoding property is used to convert between AnsiString and Unicode strings.

So, in your case, when you send a native String, Indy will first convert the AnsiString to Unicode using the DefAnsiEncoding, then convert the resulting Unicode to bytes using the DefStringEncoding, and then finally send the resulting bytes.

By default, DefStringEncoding is set to IndyTextEncoding_ASCII and DefAnsiEncoding is set to IndyTextEncoding_OSDefault. So, any non-ASCII characters will get converted to Unicode U+003F, and thus to byte 0x3F.

In your case, you are applying the 8bit encoding only to DefStringEncoding (the final Unicode->bytes conversion), but you are likely losing data from DefAnsiEncoding (the initial AnsiString->Unicode conversion), if you try to send ANSI characters that are outside of the user's locale.

If you want to send an AnsiString as-is without any conversions, you would need to set both the DefAnsiEncoding and the DefStringEncoding properties to the same encoding, effectively cancelling them out, and thus sending the AnsiString's raw bytes as-is. Provided, of course, that the encoding of the AnsiString's characters actually matches whatever encoding you set the DefAnsiEncoding to, or else you will risk losing data.

Setting both properties to the 8bit encoding is an option (but I don't recommend it, as you will run into problems if the client and server are operating with different locales).  I suggest you leave the DefAnsiEncoding set to IndyTextEncoding_OSDefault (and don't send any characters that are outside of the user's locale), and set the DefStringEncoding to IndyTextEncoding_UTF8.  Alternatively, set both properties to IndyTextEncoding_UTF8, and then make sure any AnsiString you want to send holds UTF-8 data instead of ANSI data.

Whatever you decide, you will have to replicate the same setup on the server side, too.  Is it also written in Delphi 5?

(01-09-2023, 03:49 AM)eddie1473 Wrote: I'm new to Delphi so that could easily be the case!

Why are you starting out with an outdated version of Delphi that is 23.5 years old, instead of using a modern, up-to-date version?

Reply
#3
Thanks rlebeau! That was a great description of what was causing the underlying issue, and setting DefAnsiEncoding solved the problem. While searching for solutions I saw references to Delphi 5 being pre-Unicode and suspected that was related but I wasn't able to put it all together, and I didn't notice the DefAnsiEncoding property.

Quote:Why are you using the 8bit encoding at all, instead of, say, UTF-8 instead?

I am working on a app that has been deployed to end users for many years and I'm just trying to tweak it to add SSL for secure communication. The app has a complex, home grown protocol for sending text and formatting codes to the server which is not something I want to mess with for now. But you are correct, I double checked the comments and it is indeed using UTF-8 so I have set both DefAnsiEncoding and DefStringEncoding to IndyTextEncoding_UTF8 which seems to work just fine.

Quote:Whatever you decide, you will have to replicate the same setup on the server side, too.  Is it also written in Delphi 5?

I have not looked at the server code at all but I believe it is written in Java. In any case, it is already setup to handle the UTF-8 text and codes sent by the client.

Quote:Why are you starting out with an outdated version of Delphi that is 23.5 years old, instead of using a modern, up-to-date version?

Believe me, I would love to work with the latest Delphi version! But this is a Delphi 5 app which is already deployed so that is where I need to start. One of the other items to address is to upgrade to a modern Delphi version but that seems like a big project so it may be awhile. If you have any suggestions/resources for upgrading an app from Delphi 5 feel free to share them! I think I read that the new versions of Delphi include Indy and the old TCP classes are deprecated, so I'm hoping that the switch to Indy is a step in the right direction in terms of upgrading (as well as adding SSL capabilities).

Thanks again for your help!

-Ed
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)