10:11 a.m.
Hi developers and enthusiasts,
A while ago I introduced myself here, as I am doing a study into user
space memory forensics.
My goal is to develop a general method to extract information from
user space, from any process. The past few months I have worked on
some proof of concepts, and some basic experimentation. This has all
been fairly easy, but I want to do something more complex. Today I am
mailing you to ask for some ideas and suggestions.
To give you an idea of what I have done so far (and the level I'm
working on), here's a short list:
- Extract register contents from the process kernel stack
- Determine the exact locations of stack frames
- Make a list of all process threads and access their stacks
- Walk a list of pointers and do heuristic search on small structs,
such as common networking structs
(NOTE: All using a linux dump)
All of this worked fairly well, and it was fun, but not exactly
groundbreaking. I thought it would be best to ask the advice from
professionals before I continue.
The direction I would like to take is doing something more foolproof
than a heuristic search, I would rather derive the location of
structures using information in memory.
One idea I had was to identify the whole stack trace (i.e. including
function names) and use this to find struct locations. Since there is
always the issue of missing symbols, I would somehow have to
reintroduce these, and also figure out how dynamic symbols work. I am
especially interested in common libraries with this approach, since
something I can apply to libraries, I can apply to many processes.
Another idea I had would be to use the source/headers of the
application, maybe similar to the way volatility uses the symbols for
the kernel.
Now I don't really have a question about this, I am just looking for
ideas. Please tell me if you think my ideas are good, or bad. If you
read this and wonder why I don't do X instead of Y, please ask me.
Feel free to mention any problems you see coming. I am really looking
for a discussion here. The only requirements I have at this moment are
that it stays roughly the same topic and it is academically relevant.
That last part also means that if you know about somebody doing
exactly this, do let me know, so I can do something else.
If you know of any technical documentation or scientific research in
the area, I'm also interested in that. The most relevant paper (in the
whole topic of user space memory forensics) I have read so far is
"Digging for Data Structures", Cozzie et al (thanks emdel for showing
me that one).
I permanently reside on IRC, Freenode/#volatility, if anyone cares to
chat some more about the topic.
Cheers,
Edwin
1:42 p.m.
Hi Edwin,
Sorry for the delay. Your research sounds very cool. I just have a few
quick suggestions. First, you may already be aware of Carl Pulley's plugins
[1] - in particular the exportstack.py and symbols.py which do similar
tasks on Windows memory dumps. It may be a useful point of reference. Also,
in the past I've mined heap headers for meta-data leading to clues on what
the heap entries may contain. Its the same basic idea as kernel pool tag
scanning etc, but in user mode memory. For example if you know the
approximate size of the structure you're looking for, obviously you can
eliminate any heap entries that are too small or large for storing them. In
the Windows world, there is a poorly documented feature called heap tagging
[2] where individual heap entries have associated tags to mark their
contents, much like pool tags. You may look around for something similar in
the Linux realm. Finally, if you have access to application symbols and the
ability to use distorm3 stream disassembler, you might look for
instructions in a binary that you know reference structures in process
memory.
Hope this helps some,
MHL
[1] https://github.com/carlpulley/volatility
[2] http://www.alex-ionescu.com/?p=18
On Fri, May 24, 2013 at 10:11 AM, Edwin Smulders
<edwin.smulders(a)gmail.com>wrote:
Hi developers and enthusiasts,
A while ago I introduced myself here, as I am doing a study into user
space memory forensics.
My goal is to develop a general method to extract information from
user space, from any process. The past few months I have worked on
some proof of concepts, and some basic experimentation. This has all
been fairly easy, but I want to do something more complex. Today I am
mailing you to ask for some ideas and suggestions.
To give you an idea of what I have done so far (and the level I'm
working on), here's a short list:
- Extract register contents from the process kernel stack
- Determine the exact locations of stack frames
- Make a list of all process threads and access their stacks
- Walk a list of pointers and do heuristic search on small structs,
such as common networking structs
(NOTE: All using a linux dump)
All of this worked fairly well, and it was fun, but not exactly
groundbreaking. I thought it would be best to ask the advice from
professionals before I continue.
The direction I would like to take is doing something more foolproof
than a heuristic search, I would rather derive the location of
structures using information in memory.
One idea I had was to identify the whole stack trace (i.e. including
function names) and use this to find struct locations. Since there is
always the issue of missing symbols, I would somehow have to
reintroduce these, and also figure out how dynamic symbols work. I am
especially interested in common libraries with this approach, since
something I can apply to libraries, I can apply to many processes.
Another idea I had would be to use the source/headers of the
application, maybe similar to the way volatility uses the symbols for
the kernel.
Now I don't really have a question about this, I am just looking for
ideas. Please tell me if you think my ideas are good, or bad. If you
read this and wonder why I don't do X instead of Y, please ask me.
Feel free to mention any problems you see coming. I am really looking
for a discussion here. The only requirements I have at this moment are
that it stays roughly the same topic and it is academically relevant.
That last part also means that if you know about somebody doing
exactly this, do let me know, so I can do something else.
If you know of any technical documentation or scientific research in
the area, I'm also interested in that. The most relevant paper (in the
whole topic of user space memory forensics) I have read so far is
"Digging for Data Structures", Cozzie et al (thanks emdel for showing
me that one).
I permanently reside on IRC, Freenode/#volatility, if anyone cares to
chat some more about the topic.
Cheers,
Edwin
_______________________________________________
Vol-dev mailing list
Vol-dev(a)volatilityfoundation.org
http://lists.volatilityfoundation.org/mailman/listinfo/vol-dev
4:51 a.m.
Hi Michael,
Thanks for your response.
I was not aware of the plugins by Carl Pulley. They include references
to the research, so that is definitely helpful.
Not so sure about pool/heap tags, I think I'll skip that for now.
I've already talked about something with Andrew W. in a private
conversation, namely the windows debug symbols. From the plugins you
linked, I gather that they are acquired from a webservice of sorts.
But the symbols.py plugin also seems to function without these debug
symbols. I found this comment:
"If use_symbols is False or no symbol information is available, then
module exports information is used to populate the symbols tables."
Since I am not very familiar with the windows world yet, do you know
what "module exports information" means?
The plugins do some more things I had hoped to do, such as finding ROP
chains. I could still implement it for linux, but it would not be all
that academically relevant anymore :)
I've had some time already after I sent this thread start, so I've
actually implemented finding the stack frames and connecting them to
function symbols (not debug symbols), using distorm3 to find the
relevant addresses. It works for my test program which calls both libc
and my own test library, so I'm pretty happy with that. Still needs a
lot of testing and polish though :)
Since it's looking more and more like my research won't be all that
new, I think I'll focus on a more broad context after this, drawing a
comparison to different compilers and options, and comparing the 4 big
operating systems (ie windows, linux, *bsd, os x).
Cheers,
Edwin
On 5 June 2013 19:42, Michael Hale Ligh <michael.hale(a)gmail.com> wrote:
Hi Edwin,
Sorry for the delay. Your research sounds very cool. I just have a few quick
suggestions. First, you may already be aware of Carl Pulley's plugins [1] -
in particular the exportstack.py and symbols.py which do similar tasks on
Windows memory dumps. It may be a useful point of reference. Also, in the
past I've mined heap headers for meta-data leading to clues on what the heap
entries may contain. Its the same basic idea as kernel pool tag scanning
etc, but in user mode memory. For example if you know the approximate size
of the structure you're looking for, obviously you can eliminate any heap
entries that are too small or large for storing them. In the Windows world,
there is a poorly documented feature called heap tagging [2] where
individual heap entries have associated tags to mark their contents, much
like pool tags. You may look around for something similar in the Linux
realm. Finally, if you have access to application symbols and the ability to
use distorm3 stream disassembler, you might look for instructions in a
binary that you know reference structures in process memory.
Hope this helps some,
MHL
[1] https://github.com/carlpulley/volatility
[2] http://www.alex-ionescu.com/?p=18
On Fri, May 24, 2013 at 10:11 AM, Edwin Smulders <edwin.smulders(a)gmail.com>
wrote:
Hi developers and enthusiasts,
A while ago I introduced myself here, as I am doing a study into user
space memory forensics.
My goal is to develop a general method to extract information from
user space, from any process. The past few months I have worked on
some proof of concepts, and some basic experimentation. This has all
been fairly easy, but I want to do something more complex. Today I am
mailing you to ask for some ideas and suggestions.
To give you an idea of what I have done so far (and the level I'm
working on), here's a short list:
- Extract register contents from the process kernel stack
- Determine the exact locations of stack frames
- Make a list of all process threads and access their stacks
- Walk a list of pointers and do heuristic search on small structs,
such as common networking structs
(NOTE: All using a linux dump)
All of this worked fairly well, and it was fun, but not exactly
groundbreaking. I thought it would be best to ask the advice from
professionals before I continue.
The direction I would like to take is doing something more foolproof
than a heuristic search, I would rather derive the location of
structures using information in memory.
One idea I had was to identify the whole stack trace (i.e. including
function names) and use this to find struct locations. Since there is
always the issue of missing symbols, I would somehow have to
reintroduce these, and also figure out how dynamic symbols work. I am
especially interested in common libraries with this approach, since
something I can apply to libraries, I can apply to many processes.
Another idea I had would be to use the source/headers of the
application, maybe similar to the way volatility uses the symbols for
the kernel.
Now I don't really have a question about this, I am just looking for
ideas. Please tell me if you think my ideas are good, or bad. If you
read this and wonder why I don't do X instead of Y, please ask me.
Feel free to mention any problems you see coming. I am really looking
for a discussion here. The only requirements I have at this moment are
that it stays roughly the same topic and it is academically relevant.
That last part also means that if you know about somebody doing
exactly this, do let me know, so I can do something else.
If you know of any technical documentation or scientific research in
the area, I'm also interested in that. The most relevant paper (in the
whole topic of user space memory forensics) I have read so far is
"Digging for Data Structures", Cozzie et al (thanks emdel for showing
me that one).
I permanently reside on IRC, Freenode/#volatility, if anyone cares to
chat some more about the topic.
Cheers,
Edwin
_______________________________________________
Vol-dev mailing list
Vol-dev(a)volatilityfoundation.org
http://lists.volatilityfoundation.org/mailman/listinfo/vol-dev
11:11 a.m.
Hey Edwin,
"Module exports information" in Windows means walking the PE export table
(a list of functions or global variables exposed for sharing). For example
kernel32.dll exports ReadFile(), WriteFile(), CreateProcess(), etc. You
don't need debug symbols to find these functions in memory because the
export table is your map/guide. However, not all functions are
exported...and that's when debug symbols come in handy.
MHL
On Thu, Jun 6, 2013 at 4:51 AM, Edwin Smulders <edwin.smulders(a)gmail.com>wrote:
Hi Michael,
Thanks for your response.
I was not aware of the plugins by Carl Pulley. They include references
to the research, so that is definitely helpful.
Not so sure about pool/heap tags, I think I'll skip that for now.
I've already talked about something with Andrew W. in a private
conversation, namely the windows debug symbols. From the plugins you
linked, I gather that they are acquired from a webservice of sorts.
But the symbols.py plugin also seems to function without these debug
symbols. I found this comment:
"If use_symbols is False or no symbol information is available, then
module exports information is used to populate the symbols tables."
Since I am not very familiar with the windows world yet, do you know
what "module exports information" means?
The plugins do some more things I had hoped to do, such as finding ROP
chains. I could still implement it for linux, but it would not be all
that academically relevant anymore :)
I've had some time already after I sent this thread start, so I've
actually implemented finding the stack frames and connecting them to
function symbols (not debug symbols), using distorm3 to find the
relevant addresses. It works for my test program which calls both libc
and my own test library, so I'm pretty happy with that. Still needs a
lot of testing and polish though :)
Since it's looking more and more like my research won't be all that
new, I think I'll focus on a more broad context after this, drawing a
comparison to different compilers and options, and comparing the 4 big
operating systems (ie windows, linux, *bsd, os x).
Cheers,
Edwin
On 5 June 2013 19:42, Michael Hale Ligh <michael.hale(a)gmail.com> wrote:
Hi Edwin,
Sorry for the delay. Your research sounds very cool. I just have a few
quick
suggestions. First, you may already be aware of
Carl Pulley's plugins
[1] -
in particular the exportstack.py and symbols.py
which do similar tasks on
Windows memory dumps. It may be a useful point of reference. Also, in the
past I've mined heap headers for meta-data leading to clues on what the
heap
entries may contain. Its the same basic idea as
kernel pool tag scanning
etc, but in user mode memory. For example if you know the approximate
size
of the structure you're looking for,
obviously you can eliminate any heap
entries that are too small or large for storing them. In the Windows
world,
there is a poorly documented feature called heap
tagging [2] where
individual heap entries have associated tags to mark their contents, much
like pool tags. You may look around for something similar in the Linux
realm. Finally, if you have access to application symbols and the
ability to
use distorm3 stream disassembler, you might look
for instructions in a
binary that you know reference structures in process memory.
Hope this helps some,
MHL
[1] https://github.com/carlpulley/volatility
[2] http://www.alex-ionescu.com/?p=18
On Fri, May 24, 2013 at 10:11 AM, Edwin Smulders <
edwin.smulders(a)gmail.com>
wrote:
Hi developers and enthusiasts,
A while ago I introduced myself here, as I am doing a study into user
space memory forensics.
My goal is to develop a general method to extract information from
user space, from any process. The past few months I have worked on
some proof of concepts, and some basic experimentation. This has all
been fairly easy, but I want to do something more complex. Today I am
mailing you to ask for some ideas and suggestions.
To give you an idea of what I have done so far (and the level I'm
working on), here's a short list:
- Extract register contents from the process kernel stack
- Determine the exact locations of stack frames
- Make a list of all process threads and access their stacks
- Walk a list of pointers and do heuristic search on small structs,
such as common networking structs
(NOTE: All using a linux dump)
All of this worked fairly well, and it was fun, but not exactly
groundbreaking. I thought it would be best to ask the advice from
professionals before I continue.
The direction I would like to take is doing something more foolproof
than a heuristic search, I would rather derive the location of
structures using information in memory.
One idea I had was to identify the whole stack trace (i.e. including
function names) and use this to find struct locations. Since there is
always the issue of missing symbols, I would somehow have to
reintroduce these, and also figure out how dynamic symbols work. I am
especially interested in common libraries with this approach, since
something I can apply to libraries, I can apply to many processes.
Another idea I had would be to use the source/headers of the
application, maybe similar to the way volatility uses the symbols for
the kernel.
Now I don't really have a question about this, I am just looking for
ideas. Please tell me if you think my ideas are good, or bad. If you
read this and wonder why I don't do X instead of Y, please ask me.
Feel free to mention any problems you see coming. I am really looking
for a discussion here. The only requirements I have at this moment are
that it stays roughly the same topic and it is academically relevant.
That last part also means that if you know about somebody doing
exactly this, do let me know, so I can do something else.
If you know of any technical documentation or scientific research in
the area, I'm also interested in that. The most relevant paper (in the
whole topic of user space memory forensics) I have read so far is
"Digging for Data Structures", Cozzie et al (thanks emdel for showing
me that one).
I permanently reside on IRC, Freenode/#volatility, if anyone cares to
chat some more about the topic.
Cheers,
Edwin
_______________________________________________
Vol-dev mailing list
Vol-dev(a)volatilityfoundation.org
http://lists.volatilityfoundation.org/mailman/listinfo/vol-dev
4132
days inactive
4145
days old
vol-dev@lists.volatilityfoundation.org
3 comments
2 participants
participants (2)
-
Edwin Smulders -
Michael Hale Ligh