6.0.20 appears not to work with BC++ Builder 2007

[micronav]

Hi,

We use Borland C++ Builder 2007.

We have been using 6.0.16 with great success but have recently upgraded to 6.0.20 (Professional). However when we create a new project, and activate EurekaLog it seems not to work.

The attached is a very simple form with a button which should cause a crash. I had to remove the .tds from the zip file to make it small enough.

We see the EurekaLog processing window pop-up when we build it, but when we run the application it does not catch the error. We get the Borland default exception handler instead.

Any help would be greatly appreciated.

Ken.

[Alex]

Hi.

It appears that your problem is that your application uses ExceptionExpert package. This is not supported mode.
Please, remove EurekaLog's package from your run-time packages list. ExceptionExpert is an IDE expert, it is not designed to be used as run-time package.

[Alex]

P.S.
May be you can find this article interesting: http://support.eurekalog.com/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=20

[micronav]

Alex,

Thank you for the reply, that did indeed fix our problem.

Unfortunately we have encountered a second problem and I would be very grateful if you could help us.

Attached is a small project with one .exe linked to one .dll. For our current system we need to have Dynamic RTL off and Runtime packages off. This is how the attached project is configured. But when I run the project it just hangs and goes no further. Turning either option back on again causes the project to work successfully.

Can you shed any light on this?

Again, any help is very much appreciated.

Kind Regards,
Ken.

[Alex]

Hi,

That is very strange. The attached compiled exe file indeed hangs on startup. But if I try to recompile your example - everything works fine.
Can you retest it with the latest RC version of EurekaLog?
Can you reproduce this issue under debugger? If not, you can compile project with tds info and attach a debugger to hanged application to see its call stack.

[micronav]

Alex,

I've installed the latest .21_RC3 and the project still hangs. It hangs in the debugger as well. I've attached a screenshot of the callstack. I hope this helps.

Kind regards,

Ken.

[Alex]

This looks like the current thread hangs, because some other thread holds a lock on memory manager.
Is this the only thread in your hanged application? What are call stacks of others threads?
Can you also check this issue on any other machine?

P.S.
BTW, you can select all items in call stack and use Ctrl + C to copy them to clipboard.

[Alex]

Hi.

I remembered now that I've seen a very similar situation before.
It was a memory-corruption bug in customer's code. This bug corrupted memory manager's internal structures and there was access violation inside MM's functions, when EurekaLog tried to process another exception. I don't remember the exact details, but the symptoms were very similar: very strange hung inside GetMem and normal work without EurekaLog.
I suggest you to turn on Range Check errors, catching mem-leaks in EurekaLog and use full debug mode in FastMM.

See also:
http://blog.eurekalog.com/?p=198
http://blog.eurekalog.com/?p=177
http://blog.eurekalog.com/?p=135
http://blog.eurekalog.com/?p=122

[micronav]

Alex,

First to answer your questions:

Yes this is the only thread.
There are no other call stacks.
And I have rebuilt the project on another machine with the same result.

I understand what you are saying about the GetMem issue but you can see the code we are trying to test. It's very simple. Just a single exported function from one DLL which is imported into the EXE.

I turned on Range Checking and Overflow Checking but it made no difference. Unfortunately I cannot test the mem-leaks as the option is greyed out. I've done a little research and as far as I know Builder 2007 uses the FastMM by default.

I appreciate this isn't much help. Do you have an alternate FTP site? I could provide you all the tds files if that would help?

Regards,

Ken.

[Alex]

Hi.

Ok, I wanted to make sure, that your attached example is exact one. I'll try to dig deeper into your demo then.
BTW, you can try to install latest updates on C++ Builder itself to see if it helps.

[micronav]

Hi Alex,

Any news on this?

Just to let you know, we already have the latest updates for Builder 2007 so it would appear there is no change.

Look forward to hearing from you.

Ken.

[micronav]

Alex,

Is anything happening regarding this bug?

We cannot use EurekaLog because of it.

[Alex]

Hi,

I'm really sorry for this long delay, but we're still working on it.
I hope to give some news soon.

[micronav]

Hi Alex,

Well I decided to break out the debugger and after a 5 hour session I have some information for you which may help find the issue:

EurekaLog makes 2 calls to AddModuleFromFilename first for Project1.dll and second for Project2.exe. The first call succeeds but the second fails.

The ZDecompress routine in EZLib2.pas seems to be at fault. There is a GetMem line (~line 299) which is subsequently reallocated (~line 315). The second time around its the ReAllocate which is failing. I think the inflate() call might be corrupting the memory somewhere.

What I think might be happening is the InflateInit might be allocating some memory on the first call which it then relies on for the second. But with DynamicRTL turned off, the second call is using a different memory manager, and could possibly be attempting to modify pointers it doesn't own.

It's just a guess at the moment.

If there was any way you could provide the .pas files for me I will gladly debug the issue and provide you with a fix.

I'll try and look into it further but its hard to do without the pas files :)

I hope this helps a bit anyway,

Kind regards,
Ken.

[Alex]

The ZDecompress routine in EZLib2.pas seems to be at fault. There is a GetMem line (~line 299) which is subsequently reallocated (~line 315). The second time around its the ReAllocate which is failing. I think the inflate() call might be corrupting the memory somewhere.

Yes, that is quite close to truth. The problem arise in the inflate call. Inflate calls GetMem to allocate an internal buffer. The real problem is that this GetMem call returns "invalid pointer" for some reason, which overlaps with real used memory (I haven't debugged it yet). So, inflate writes data into that buffer (which it thinks is valid), but really it corrupts some other valid memory.

At my machine inputs for inflate are:
inputbuf = $004C15AB
inputbufsize = $6AE
outputbuf = $00B45780
outputbufsize = $700

inflate calls GetMem exactly 1 time:
size = $2530
result = $00580724 // <- that is wrong one (see below)

Corrupted data lies near:
headerdata1 = $00581244
headerdata2 = $00582244

Corrupted data represents FastMM's MediumBlockBins static array. Obviosly, its address can not be changed during run. Once corrupted, MediumBlockBins gaves an AV at ReallocMem right after inflate's call.

Note, that the buffer, returned by GetMem, ($00580724) overlaps with MediumBlockBins's data: $00581244 - $00580724 = $B20 < $2530.

So, it's not surprising that data became corrupted.

Now, I need to figure out, why GetMem returns this obviosly invalid address.

The hang itself occurs because FastMM's code is not protected against AV in itself. I.e. there are no try/finally calls (probably for speed reason), so if there is AV inside MM's code, the MM can be locked forever. And the very next request to MM will block your application. That is the reason for hung application.

If there was any way you could provide the .pas files for me I will gladly debug the issue and provide you with a fix
Unfortunately, it is not up to me to make such decision :( I'll ask Fabio about this ASAP.
Anyway, thanks for your great assistance!

[micronav]

Alex,

I wonder if this is a linker issue?

With DynamicRTL turned off there should effectively be 2 MM's... one in Project1.dll and one in Project2.exe.

I wonder if the linker is linking the Project2.exe GetMem against the Project1.dll? If it did the address space would be wrong and you would get the invalid address.

Just a thought.

If there was any way you could provide the .pas files for me I will gladly debug the issue and provide you with a fix
Unfortunately, it is not up to me to make such decision I'll ask Fabio about this ASAP.

That's ok. I understand it might not be possible :)

I'll take another look with the information you gave me and perhaps I'll hit on something.

Best regards,
Ken.

[Alex]

Hi,

I wonder if the linker is linking the Project2.exe GetMem against the Project1.dll? If it did the address space would be wrong and you would get the invalid address.
Well, in this case a second copy of MM will be unable to allocate memory, which is occupied by first MM, right? I.e. first copy of MM calls VirtualAlloc and it returns memory block. The second copy calls VirtualAlloc and get a new block. These two blocks can not intersect.
Or do I not understand you?

In any case, I tried to find a source of this inproper GetMem behavior and found a VERY strange thing.
It appears that GetMem returns pointer to his internal structure instead of actual memory block, associated with this structure!
Namely, GetMem returns pointer to element of MediumBlockBins array.

May be it is just me, that I don't understand something, but it looks somehow wrong.

Here is my tracing of problem call (my input numbers are listed in prev. post):

function SysGetMem(Size: Integer): Pointer;
asm
// Enter SysGetMem, eax = $2530

{On entry:
eax = ASize}
{Since most allocations are for small blocks, determine the small block type
index so long}
lea edx, [eax + BlockHeaderSize - 1] // BlockHeaderSize = 4, edx = $2533
shr edx, 3 // edx = $4A6
{Is it a small block?}
cmp eax, (MaximumSmallBlockSize - BlockHeaderSize) // right part = $A2C, eax = $2530
{Save ebx}
push ebx // ebx = $0012FD3C
{Get the IsMultiThread variable so long}
mov cl, IsMultiThread // @IsMultiThread = $0057fae5, cl = 1
{Is it a small block?}
ja @NotASmallBlock // ok, go

@NotASmallBlock:
cmp eax, (MaximumMediumBlockSize - BlockHeaderSize) // right part = $40A2C, eax = $2530
ja @IsALargeBlockRequest // skip
{Get the bin size for this block size. Block sizes are
rounded up to the next bin size.}
lea ebx, [eax + MediumBlockGranularity - 1 + BlockHeaderSize - MediumBlockSizeOffset] // right part = [eax + D3], ebx = $2603
and ebx, -MediumBlockGranularity // right part = $FFFFFF00, ebx = $2600
add ebx, MediumBlockSizeOffset // right part = $30, ebx = $2630
{Do we need to lock the medium blocks?}
test cl, cl // cl = 1
jnz @LockMediumBlocks // ok, go

@LockMediumBlocks:
mov eax, $100 // eax = $100
{Attempt to lock the medium blocks}
lock cmpxchg MediumBlocksLocked, ah // @MediumBlocksLocked = $005801BC
je @MediumBlocksLocked // ok, go

@MediumBlocksLocked:
{Get the bin number in ecx and the group number in edx}
lea edx, [ebx - MinimumMediumBlockSize] // right part = [ebx-$b30], ebx = $2630, edx = $1B00
mov ecx, edx // ecx = $1B00
shr edx, 8 + 5 // edx = 0
shr ecx, 8 // ecx = 1B
{Is there a suitable block inside this group?}
mov eax, -1 // eax = $FFFFFFFF
shl eax, cl // eax = $F8000000
and eax, dword ptr [MediumBlockBinBitmaps + edx * 4] // right part = [$005801CC + 0 * 4], it's value = 0, eax = 0
jz @GroupIsEmpty // ok, go

@GroupIsEmpty:
{Try all groups greater than this group}
mov eax, -2 // eax = $FFFFFFFE
mov ecx, edx // ecx = 0
shl eax, cl // eax = $FFFFFFFE
and eax, MediumBlockBinGroupBitmap // right part = [$005801C8], value = $08000010, eax = $08000010
jz @TrySequentialFeedMedium // skip
{There is a suitable group with space: get the bin number}
bsf edx, eax // edx = 4
{Get the bin in the group with free blocks}
mov eax, dword ptr [MediumBlockBinBitmaps + edx * 4] // right part = [$005801CC + 4 * 4], value = $08000000, eax = $08000000
bsf ecx, eax // ecx = $1B
mov eax, edx // eax = 4
shl eax, 5 // eax = 80
or ecx, eax // ecx = $9B
jmp @GotBinAndGroup // ok, go

@GotBinAndGroup:
{ebx = block size, ecx = bin number, edx = group number}
push esi // esi = $004C15AB
push edi // edi = $700
{Get a pointer to the bin in edi}
lea edi, [MediumBlockBins + ecx * 8] // right part = [$0058024C + $9B * 8], value = $00580724, edi = $00580724
{Get the free block in esi}
mov esi, TMediumFreeBlock[edi].NextFreeBlock // esi = $00580724
{Remove the first block from the linked list (LIFO)}
mov eax, TMediumFreeBlock[esi].NextFreeBlock // eax = $00580724
mov TMediumFreeBlock[edi].NextFreeBlock, eax // does not change memory
mov TMediumFreeBlock[eax].PreviousFreeBlock, edi // does not change memory
{Is this bin now empty?}
cmp edi, eax // edi = eax = $00580724
jne @MediumBinNotEmptyForMedium // skip
{eax = bin group number, ecx = bin number, edi = @bin, esi = free block, ebx = block size}
{Flag this bin as empty}
mov eax, -2 // eax = $FFFFFFFE
rol eax, cl // ecx = $9B, eax = $F7FFFFFF
and dword ptr [MediumBlockBinBitmaps + edx * 4], eax // [$005801CC + 4 * 4], value before = $08000000, value after = 0
jnz @MediumBinNotEmptyForMedium // skip
{Flag the group as empty}
btr MediumBlockBinGroupBitmap, edx // [$005801C8], edx = 4, value before = $08000010, value after = $08000000
@MediumBinNotEmptyForMedium:
{esi = free block, ebx = block size}
{Get the size of the available medium block in edi}
mov edi, DropMediumAndLargeFlagsMask // edi = $FFFFFFF0
and edi, [esi - 4] // right part = [$00580724 - 4], value = $0058071C, edi = $00580710 <--- seems wrong! This is not the size, it's some pointer in MediumBlockBins's array
{Get the size of the second split in edx}
mov edx, edi // edx = $00580710
sub edx, ebx // ebx = $2630, edx = $0057E0E0 - a very strange value for size!
jz @UseWholeBlockForMedium // skip
{Split the block in two}
lea eax, [esi + ebx] // esi = $00580724, ebx = $2630, eax = $00582D54
lea ecx, [edx + IsMediumBlockFlag + IsFreeBlockFlag] // right part = [$0057E0E0 + 3], ecx = $0057E0E3
mov [eax - 4], ecx // eax = $00582D54, value before = 0 (0s are all around), value after = $0057E0E3
{Store the size of the second split as the second last dword}
mov [eax + edx - 8], edx // eax = $00582D54, edx = $00570E0E, left part = [$00B00E2C] value before = forget to take a look, seems random data all around, value after = $00570E0E
{Put the remainder in a bin}
cmp edx, MinimumMediumBlockSize // right part = $B30
jb @GotMediumBlockForMedium // skip
call InsertMediumBlockIntoBin // ok, call
jmp @GotMediumBlockForMedium // ok, go

@GotMediumBlockForMedium:
{Set the size and flags for this block}
lea ecx, [ebx + IsMediumBlockFlag] // ebx = $2630, ecx = $2632
mov [esi - 4], ecx // $esi = $00580724 - plain wrong! This is address of 1 element in MediumBlockBins array, value before = $00580710, value after = $2632
{Unlock medium blocks}
mov MediumBlocksLocked, False // ok, the rest doesn't matter...
mov eax, esi
pop edi
pop esi
pop ebx
ret
end;

Notice, how esi gets its value in "@GotBinAndGroup:" block. That is a member of MediumBlockBins array, yet it don't ever changes later. And it will be a returning value for SysGetMem's result.
Another strange thing is code in "@MediumBinNotEmptyForMedium:" block. This code threats ESI as some kind of other structure. Namely - with Size field. But TMediumFreeBlock structs do not have Size field. So this code ends up corrupting MediumBlockBins array instead.

[micronav]

Wow!

You've been busy!

I'll have to take this home and digest it. Unfortunately I won't have much time tonight but I'll get back to it tomorrow night.

Here's another possibility for you though:

Is EZlib2 compiled against the dynamic MM? If it was it might be compiled for the dynamic MM and linked against the static version. Remember, I only have .obj files, not the pas files so it would not re-compile it when I turned off DynamicRTL. If you have the pas files on your system it might explain why a recompile might make the problem go away.

Again, just another shot in the dark.

Thank you for looking into it so deeply. I'll try and catch up with all the info tomorrow night.

Best regards,
Ken.

[micronav]

Hi Alex,

Well I looked at this again last night. I managed to get to the same point as you did just prior to the inflate call:

inputbuf = $004C15AB
inputbufsize = $6AE
outputbuf = $00B45780
outputbufsize = $700

.. but I couldn't catch the GetMem in inflate. It's almost impossible to do just using the CPU window.

I'm afraid I've done as much as I can at the moment. Have you spoken to Fabio about the .pas files? I'd really like to help but it's proving too hard with just the .obj files.

Regards,
Ken.

[admin]

Hi micronav,

I REALLY appreciate all your efforts done on this issue identifications so I have decided to change your Company license from Professional to Enterprise (with full sources) for free! :)

Re-download your EurekaLog version and you will obtain full sources! :)

I hope that can work directly on the sources can definitely speedup our cooperation on this bug solution! ;)

[micronav]

Sweet! :) Thank you!

I'll download those and test as soon as I can.

Regards,
Ken.

[admin]

Hi Ken,

you're welcome! :)

[Alex]

Hi,

I've asked about this issue on FastMM's forum, see here: https://sourceforge.net/forum/message.php?msg_id=7525596

And Pierre le Riche provided a special version of FastMM with additional checks.

I was able to track down the problem with its help, but not sure, how to solve it.

The problem is that exe file imports almost every function from DLL. I.e. when you added EurekaLog to exe, only ExceptionLog unit is added to exe. C++ Builder's linker takes all other code from DLL.

So, basically, code in exe mixes calls to exe and DLL. Since there are 2 MM available (1 for exe and 1 for DLL), then it is not surprising that there are problems here, as most data exchanges between 2 MMs (due to mixed code). And this leads to all kind of bad things.

So, the reason is that linker takes EurekaLog's code from DLL, not from exe.
Unfortunately, I'm not much of C++ Builder expert so I'm unable to tell, why is it so. Probably, functions are added to Project1.lib, that's why linker imports them from DLL?

Do you have any ideas?

[micronav]

Hi Alex,

Good work on tracking the problem down! I suspected it was something like this but wasn't sure where to look.

I haven't had much time to look at it myself and I was struggling with tracking the problem through the MM.

What you've suggested sounds exactly the issue. I'll have to look into the project options and investigate to see if there is a way we can make C++ Builder link properly.

I'll try and have a look over the weekend and get back to you.

Regards,
Ken.

[bbrandt]

Try creating a .def file that lists the few functions that you want to import from your dll into your exe. Run "implib newImportLibName.lib yourDefFileName.def" to generate a new import library. This way you will not be importing every VCL function from your dll.

I have had numerous issues using the compiler generated import libraries. In most cases this is what I do to workaround the issue.

More info on creating def files:
http://msdn.microsoft.com/en-us/library/d91k01sh(VS.80).aspx
Import by name, not by ordinal unless you know a way to keep the compiler from changing them.

[Alex]

Hi,

bbrandt, thanks for sharing your expierence!

micronav, is there any news? :)