Atozed Forums

I had a serious problem with iconv encoding library on Android 10 application developed with LAMW and Indy.
I received SIGEGV Exception with backtrace:

...
01-28 16:05:24.303  2648  2648 F DEBUG   : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
01-28 16:05:24.303  2648  2648 F DEBUG   : Cause: null pointer dereference
01-28 16:05:24.303  2648  2648 F DEBUG   :     r0  e9595fb4  r1  00000000  r2  00000000  r3  00000000
01-28 16:05:24.303  2648  2648 F DEBUG   :     r4  e9595f80  r5  ecc07048  r6  ed0fb200  r7  ed0fb200
01-28 16:05:24.303  2648  2648 F DEBUG   :     r8  c4e2efe8  r9  ed0f3160  r10 c4e093bc  r11 c38b871c
01-28 16:05:24.303  2648  2648 F DEBUG   :     ip  c38b86bc  sp  c38b8600  lr  c4d8b850  pc  efe3a950
01-28 16:05:24.309  2648  2648 F DEBUG   : 
01-28 16:05:24.309  2648  2648 F DEBUG   : backtrace:
01-28 16:05:24.309  2648  2648 F DEBUG   :       #00 pc 00066950  /apex/com.android.runtime/lib/bionic/libc.so (iconv+36) (BuildId: dcf0e174e93e33d22f35a631ba9c0de5)
01-28 16:05:24.309  2648  2648 F DEBUG   :       #01 pc 0010284c  /data/app/org.teamlooktools.teamlookmobile-s_uRfmpjft4VcskmjhvlvQ==/lib/arm/libcontrols.so (BuildId: d24f8df194618599d7a0a2d459ad6f929a806d9b)


I recompile the libiconv.so library, but the problem still exists.
On Android 8-9 everything is OK.
So, I took a decision to try to eliminate  libiconv dependancy form indy code.

Please, find attached here the changes 

In short: I made a change in Idcompilerdefines with which in case of compilation for android under FPC I define USE_LCONVENC . In Idglobals.pas instead of iconv I use unit lconvencoding.

in a method 

I do conversion in new way using lconvencoding:


the similar in other method


I use:


instead of iconv.

PLease, may I kindly ask you who wants to test if everything is OK?

Edit: Now it works fine for my application. I update the files attached.
The problem under Android 10 disappear.

(01-29-2021, 09:07 AM)ZGabrovski@gmail.com Wrote: [ -> ]I had a serious problem with iconv encoding library on Android 10 application developed with LAMW and Indy.
I received SIGEGV Exception with backtrace:

Is it because Indy is using iconv incorrectly on Android?  Or that iconv itself is just buggy on Android?

(01-29-2021, 09:07 AM)ZGabrovski@gmail.com Wrote: [ -> ]So, I took a decision to try to eliminate  libiconv dependancy form indy code.
...
PLease, may I kindly ask you who wants to test if everything is OK?

I will continue to review it more deeply as time permits, but some things I notice so far:

- your copy of IdCompilerDefines.inc and IdGlobal.pas are a little out of sync with Indy's latest code on GitHub.

- your use of {$IF} won't work for Indy, since Indy still supports compiler versions that don't support {$IF}.  But that is minor thing to fix.

- Is the LConvEncoding unit available on all platforms that FPC supports?  If so, it would make sense to just enable it unconditionally when compiling with FPC.  Except, I'm not overly thrilled with the idea of using an intermediate UTF-8 conversion (UTF-16 <-> UTF-8 <-> charset), that just seems like more work, when iconv can handle conversions to/from UTF-16 directly (UTF-16 <-> charset). But, if that is what we have to do to remove the dependancy, so be it. Just hope the performance overhead will be acceptable.

- I'm noticing areas of IdGlobal.pas that use {$IF(N)DEF USE_ICONV} were not updated to account for USE_LCONVENC.

(01-29-2021, 09:07 AM)ZGabrovski@gmail.com Wrote: [ -> ]In short: I made a change in Idcompilerdefines with which in case of compilation for android under FPC I define USE_LCONVENC.

{$IFDEF ANDROD} <- typo on your part, FPC does not define it that way. There is some other code in Indy that already handles ANDROID special.

- your copy of IdCompilerDefines.inc and IdGlobal.pas are a little out of sync with Indy's latest code on GitHub.

I took Indy form Online Package Manager in Lazarus. It is not a proble to update last trunk.
LconvEncoding should work for all fpc versions.

I will kindly ask more people to participate and test with there applications (all *nix platforms, not only android). Yesterday I did peace of code that compares the result of conversion between Lazarus LIconvEncoding functions and iconv - in case of my application they are identical.
It is not depend from the way indy call iconv. You call in the same way in 8,9,10. But in 10 it crash. I spent two days to recompile libiconv.so with the same NDK that I am using for the application. Nothing. Again crash. 

How indy crash in Android 10 and Lazarus

(01-30-2021, 07:02 AM)ZGabrovski@gmail.com Wrote: [ -> ]I took Indy form Online Package Manager in Lazarus. It is not a proble to update last trunk.

Sounds like OPM needs to be updated then. I thought the OPM maintainer was already keeping Indy in sync with new updates? Oh well, I have just pinged him.

(01-30-2021, 07:02 AM)ZGabrovski@gmail.com Wrote: [ -> ]LconvEncoding should work for all fpc versions.

I'm not so worried about VERSIONS as I am about PLATFORMS. Before I consider incorporating LConvEncoding into Indy for platforms other than Android, I would need to know if it actually works on other platforms besides just 'Nix.

More importantly, what about charsets that LConvEncoding doesn't support? Indy would have to delegate to SOMETHING for them. Looking at the source code for LConvEncoding.pas, it looks like it only implements a handful of Windows-125x, ISO-8859-x, and a few other charsets, nowhere near the number of charsets that iconv implements. It appears that LConvEncoding delegates to iconv for all other charsets. So how is that any different than Indy using iconv? Indy is simply using FPC's own iconv wrapper when compiling with FPC. So I would expect LConvEncoding to crash on charsets other than the ones that it manually implements.

I had considered updating Indy to implement common charsets directly in Indy's code. But that would take time and effort to add, and doesn't address the issue of unsupported charsets, either.

(01-30-2021, 07:02 AM)ZGabrovski@gmail.com Wrote: [ -> ]It is not depend from the way indy call iconv. You call in the same way in 8,9,10. But in 10 it crash.

Indy 8 and 9 did not use iconv at all. Support for iconv is only in Indy 10 (added/updated within the last 10 years). So again, what EXACTLY is crashing? Your earlier trace only provides one useful piece of information ("Cause: null pointer dereference"), but doesn't explain WHERE the null pointer is being used. I would rather see that error fixed if possible, before updating Indy to use a completely different charset library (especially one that will REDUCE functionality.

(01-30-2021, 07:02 AM)ZGabrovski@gmail.com Wrote: [ -> ]I spent two days to recompile libiconv.so with the same NDK that I am using for the application. Nothing. Again crash. 

Did you try debugging into Indy's/iconvenc's source code to figure out where the crash was actually occurring, and under which conditions exacty?

(01-30-2021, 07:02 AM)ZGabrovski@gmail.com Wrote: [ -> ]How indy crash in Android 10 and Lazarus

The only useful thing that came out of that discussion (and as you may notice, I was involved in that discussion), is that FPC's iconvenc unit is not compatible with Android API 28 (Android 9). That is not really an Indy issue, but a FPC issue.

"Indy 8 and 9 did not use iconv at all. Support for iconv is only in Indy 10 (added/updated within the last 10 years). So again, what EXACTLY is crashing? Your earlier trace only provides one useful piece of information ("Cause: null pointer dereference"), but doesn't explain WHERE the null pointer is being used. I would rather see that error fixed if possible, before updating Indy to use a completely different charset library (especially one that will REDUCE functionality."

I am talking for Android 8,9,10, INdy is always 10
Many peoples reported for the problems with Android 10 and libc (not for indy and Iconv).
I am using ndk r18, I recompiled iconv with the same ndk, the result is the same - craash.

"Did you try debugging into Indy's/iconvenc's source code to figure out where the crash was actually occurring, and under which conditions exacty?"

Condition is one and the same - first call of iconv.
But it is not Indy problem - I try to call Iconv outside the indy in a simple application, the result is exactly the same - crash.

"More importantly, what about charsets that LConvEncoding doesn't support? Indy would have to delegate to SOMETHING for them."

In that cases I just copy input as output.

I know that proposed solution is not the best one, thats the reason that I propose it as a "bypassing" of the problem in Android. Now, nobody can not use indy in Android 10.

(01-30-2021, 07:02 AM)ZGabrovski@gmail.com Wrote: [ -> ]I took Indy form Online Package Manager in Lazarus. It is not a proble to update last trunk.

I have confirmed that the version of Indy in OPM is up-to-date with Indy's GitHub repo. The changes you provided earlier were to a copy of Indy that was not the latest from OPM or GitHub. That is OK, the changes are isolated enough that I can review and consider incorporating them, but I'm just making you aware that the copy of Indy you are working with is not up-to-date.

(01-31-2021, 08:16 AM)ZGabrovski@gmail.com Wrote: [ -> ]I am talking for Android 8,9,10, INdy is always 10

Oh, OK. So, iconv works on Android prior to v10, just not on v10?

(01-31-2021, 08:16 AM)ZGabrovski@gmail.com Wrote: [ -> ]Many peoples reported for the problems with Android 10 and libc (not for indy and Iconv).
I am using ndk r18, I recompiled iconv with the same ndk, the result is the same - craash.

(01-31-2021, 08:16 AM)ZGabrovski@gmail.com Wrote: [ -> ]Condition is one and the same - first call of iconv.
But it is not Indy problem - I try to call Iconv outside the indy in a simple application, the result is exactly the same - crash.

Then you should be able to more easily debug exactly why it is crashing. What does your test code look like? What is the exact error message? Your earlier post shows the error as a "null pointer dereference", so WHERE is the null pointer exactly?

Test code - very simple - just call function

function Iconvert(S: string; var Res: string; const FromEncoding, ToEncoding: string): cint;
from iconvenc unit on button click.

Something like

uses iconvenc;
procedure testme;
var S : UnicodeString;
begin
Iconvert('dfsfsdf',S,'UTF8','UTF16')
end;

it is failed somewhere in iconv(H, nil, nil, @Dst, @Outlen);

But how to debug inside iconv?
Believe me I spent a hours to build the library with the same ndk version that I am using for my project.
The result is the same and the same.

(02-01-2021, 07:50 AM)ZGabrovski@gmail.com Wrote: [ -> ]

Code:

function Iconvert(S: string; var Res: string; const FromEncoding, ToEncoding: string): cint;
from iconvenc unit on button click.

Note that the function is using String parameters, even for the output data. Are you compiling with {$ModeSwitch UnicodeStrings} or {$Mode DelphiUnicode} enabled to make String=UnicodeString? I ask because...

(02-01-2021, 07:50 AM)ZGabrovski@gmail.com Wrote: [ -> ]

Code:

uses iconvenc;
procedure testme;
var S : UnicodeString;
begin
Iconvert('dfsfsdf',S,'UTF8','UTF16')
end;

You are passing a UnicodeString variable to a var String parameter, which I would not expect to compile when String=AnsiString.

Yes, UnicodeString uses UTF-16, but I would have expected the 1st and 2nd parameters of Iconvert() to be untyped to allow different kinds of strings depending on the values of the FromEncoding and ToEncoding parameter.

But, in any case, Indy doesn't use Iconvert() at all, it uses iconv() directly. And appears to be using iconv() slightly differently than Iconvert() does (I think Indy's way is correct).

(02-01-2021, 07:50 AM)ZGabrovski@gmail.com Wrote: [ -> ]it is failed somewhere in        iconv(H, nil, nil, @Dst, @Outlen);

That looks like a perfectly valid call to iconv() to me, per the iconv documentation:

Quote:The main case is when inbuf is not NULL and *inbuf is not NULL... (doesn't apply here)

...

A different case is when inbuf is NULL or *inbuf is NULL, but outbuf is not NULL and *outbuf is not NULL. In this case, the iconv function attempts to set cd’s conversion state to the initial state and store a corresponding shift sequence at *outbuf. At most *outbytesleft bytes, starting at *outbuf, will be written. If the output buffer has no more room for this reset sequence, it sets errno to E2BIG and returns (size_t)(−1). Otherwise it increments *outbuf and decrements *outbytesleft by the number of bytes written.

A third case is when inbuf is NULL or *inbuf is NULL, and outbuf is NULL or *outbuf is NULL... (shouldn't apply here)

Though, that does beg the question, where is @Dst pointing to exactly, and is it valid? Looks like it is pointing at the internal character buffer of your output UnicodeString, but Iconvert() treats that parameter as a String and attempts to resize it before calling into iconv(). Maybe that is resulting in bad memory in your case?

I do question why Iconvert() is making that particular call to iconv(), since it is being used to reset iconv's conversation state AFTER a conversion has been performed, which seems wrong or at least redundant to me.

(02-01-2021, 07:50 AM)ZGabrovski@gmail.com Wrote: [ -> ]But how to debug inside iconv?

You don't think you need to debug inside of iconv() itself, you just need to debug the code that is calling iconv().

(02-01-2021, 07:50 AM)ZGabrovski@gmail.com Wrote: [ -> ]Believe me I spent a hours to build the library with the same ndk version that I am using for my project.
The result is the same and the same.

I think you are focusing on the wrong thing. You should not be focusing on iconv itself, but rather on how it is being used by FPC/Indy.

OK, but why this code has no problem with android 8/9, only fails with Android 10?