stack getting clobbered (value of esp)

[jorgander]

i wrote a recursive function to search a tree structure, and i'm having problems with the code. the value of esp is not getting preserved over the recursive calls, but the strange thing is that as far as i can tell, upon every function entrance and corresponding function exit, the value is the same (i.e., i'm popping as much as i'm pushing, etc.).

is there something i'm forgetting about calling or returning from functions? are there any other things having to do with esp that occur when calling/returning from functions besides a push/pop of the caller's ip?

 

[lionelhill]

You'll have to post the code, but here's some for starters:
(1) If the routine is recursive, then esp will be different on each call, because each call will go deeper and deeper into the stack.
(2) The size of the address you have on the stack depends on whether you are calling far or near, and on 32-bit or 16-bit offset.
(3) Yes, functions often do more than just push the address. If you want local variables then you will normally push bp (ebp) and set ebp to esp so as to address the local variables relative to ebp. If you passed parameters, this can be done by stack too. But if you're in assembler you know what you're passing and how...

 

[almahtar]

Lionelhill's probably right: you'd probably best post your code for quickest response.


--------
It is an honor to recieve an honest reply. (Prov 24:26)

 

[jorgander]

i would have posted the code in the first place but it is kind of long.

obviously there is something i'm not doing right, i just can't tell what it is. but as long as i know that nothing else *should* be happening to the stack, then i know it's a bug in my code and i can eventually find it, as opposed to it something happening i didnt realize (i.e. somethning that occurs with ebp, esp, recursive functions, etc.).

 

[jorgander]

something very strange when i'm debugging with msvc++ 6...

if i step into every function call, esp doesn't get screwed up. however, if i step over a function call, it gets changed.

to be more detailed, there are two sections of the recursive function, one to handle leaf nodes and one to handle parent nodes. there is a jump that moves to different parts of the code depending on parent/leaf status. i can step over calls that would result in a leaf node being processed with esp remaining the same, but if i step over the a call that would process a parent, esp is wasted. in fact, it gets set to the what esp was when the last leaf node returned (that is, what esp was at the end of the recursion). mind you, this only happens if i step over parent node calls. if i step into and through it, esp is OK.

basically i can't tell why esp is getting clobbered, since it's fine if i step through it with the debugger.

 

[lionelhill]

I'm deeply suspicious about this, because things nearly always run with the debugger exactly as they would without. But there is one exception (probably more! There's one exception I know of!)
If you (ill-advisedly!) do a conditional jump by rewriting the next instruction (i.e. self-modifying code) it will work find in a debugger but probably not in "real life", where the instruction may already be in the prefetch queue before it got modified.
But I very much doubt this is your problem.

 

[almahtar]

....demon posession?

--------
It is an honor to recieve an honest reply. (Prov 24:26)

 

[oleksii]

defenitely we need a least the proc entrance/exit code to look at for making conclusions. little comment to lionehill - as far as i know any jump (in fact, any instruction changing IP) invalidates the prefetch queue. at least the things were this way for i486

regards

 

[lionelhill]

oleksii, you're probably right. I'm remembering reading something a long time ago, possibly pre 486, in a dubious chapter on hints how to make your program hard to disassemble (a fairly futile thing to do anyway. How many of us will ever write something that anyone would actually want to disassemble?)

 

[oleksii]

to lionelhill. probably u'll code a crackme one day?
the stuff u mentioned probably concerns jumps to a "misaligned" instruction - there is really such a antidebugging trick. concerning the prefetch queue - it's absolutely transparent on the coder side, the only thing one can bother is speed optimization: less jumps, faster execution, at least in theory..

i wonder if jorgander has found the answer?

 

[lionelhill]

Thanks for info...

 

[jorgander]

sorry i haven't been on this forum in a while...

yes, i solved the problem, and sadly it was my own naivette. u see, i am used to debugging non-assembler code with msvc++, and in the debugger when u are running a recursive function and u "step over" a function call, the debugger will stop again when ***that particular instance*** of the recursive function returns. in assembler code, however, where the debugger does not have access to the high-level code, it cannot resolve complicated things like remembering which instance of a recursive function call to stop at, etc. (of course, i'm assuming these things about msvc++'s debugger here).

so, since i was debugging a recursive function that was written to search a tree structure, the debugger was stopping at the ***most recent instance*** of the instruction, not excluding those times the instruction was called in other instances of the same function.

in short, i was assuming that the debugger was stopping at the same level in the tree hierarchy, when in fact it was stopping much farther down (at the bottom, in fact). *that* is why i thought esp was not getting preserved over function calls.

 

[lionelhill]

Thanks for telling us the answer. It is always good to remember things like that ready for when it happens to me, too.