12:22 p.m.
Hello,
With quite a bit of help & patience from ikelos, I have created a
skeleton for generic Linux support for Vol. In order to show the
complete process of how it currently works, I am going to demonstrate
the process from start to finish (a simple process listing of a memory
image). The main reason I am sending this email is because I hope
that the volatility developers can review what I have so far, comment,
and ultimately we can finalize a skeleton that can then be rapidly
added on to develop tens of useful plugins for Linux memory analysis.
I already have code for quite a bit of kernel analysis, so my whole
goal of this is to fit it inside the framework and ultimately get it
accepted into Volatility stable.
So the process of my system currently works like this (will be
illustrated with code and examples after):
1) Gather all of the structure offsets from a kernel through an LKM
2) Feed the LKM's output to a python script that generates a vtype /
profile automatically based on structure definitions kept in another
file
3) Make a profile python file for these types
4) Now we have a profile for the specific type and plugins can use it as normal
after this point we can run plugins, so now to illustrate all of this:
1) Gathering of offsets is facilitated in this LKM. (I added a simple
license & author header to each file due to ikelos's advice to do so)
http://gist.github.com/595600
This module works by gathering the offset of members from every
structure necessary to do analysis. It also adds the size of each
structure into the output file in order to have correct profile
information. Note that this code is kernel version aware and will be
able to get offsets from any 2.6.x version. Also, it only gathers a
small number of structures & members now b/c that is all that is
needed in order to list processes. Once the code is reviewed by devs,
other structures can be quickly added.
2) Feed this 1) to a python script
So at this step we need to generate the profile and we also need to
set the DTB. Even though it may seem odd, its not possible in a 2.6.x
generic way to get the cr3/dtb of the init process in order to
automatically acquite the dtb, even inside the kernel module. This is
because the DTB we want (swapper_pg_dir) is not exported and the mm
structure that references it (init_mm) stopped being exported in
2.6.26. Also, you can't get a reference to init_mm through init_task,
because after initialize init_task->mm is set to NULL. Because of all
this, we have to accept the DTB value on the commandline to this
script, but this is easily obtained by grepping for 'swapper_pg_dir'
in the corresponding System.map file for the kernel.
Here is the python script to generate the output, note that the
linux_kernel_types file will take work as well to support all
structures & members needed for full analysis, but that it will be
stable for all of 2.6.x once set:
http://gist.github.com/595589
http://gist.github.com/595598
Here is using it on the test kernel:
http://gist.github.com/595594
as you can see it has all the kernel structures and uses
volatility_magic to store the DTB. This means the users never have to
deal with DTB and it will still with the profile forever and will auto
load on vol initialization
now that we have this:
3) make a simple python profile
With the help of ikelos, I have made the profile system very easy to
add support for new kernel versions. To support this there is an
abstract Linux Profile that does everything but set the
abstract_types:
http://gist.github.com/595604
and the simple profile needed to support the debian test image:
http://gist.github.com/595606
4) time to write a plugin!
this is the simple plugin I came up with to list processes & it uses
profile just generated:
http://gist.github.com/595607
note that the system map parsing stuff will go into a utility file
once devs approve of the current way of doing things or we change
stuff around. its very quick to parse so its not an issue to parse it
on every load or eventually have the system map hash table pickled in
the cache
With that, we have distro / version generic Linux support. We also
have an easy method to generate profiles for any Linux System.
Please send any comments / suggestions, as once its approved I would
like to quickly add support for a large number of more plugins &
kernels
6:02 p.m.
I have been thinking about another issue that will need to be sorted
before we begin mass generating profiles.....
The problem is that in the current implementation, the file of
structure offsets that is used to make the profiles/vtypes is only the
offsets that the LKM is coded to gather. This is a problem because
every time we need to access a new structure type for some new
functionality, the LKM would need to be run again on all the target
systems in order to get the offsets that the new code requires. This
isn't something we can really deal with and we need a better solution.
Unfortunately the only two solutions that come to mind for this are:
1) getting all the offsets at once from a debug kernel build package
(vmlinux), but many distro vendors don't package the debug build
properly and a number of kernels get skipped
2) writing something that can scan the kernels files (not just the
header files, but all files) for a particular kernel tree and all it
would need to do is get the name of every structure and all the
members. This would also be nice b/c then instead of having to run an
LKM to get offsets, you would just have to download a particular
kernel's source package (all distros supply these) and run the script
to get all the structure names and members list. The rest of the code
could then stay how it is and we can support any access to any
structure type.
The good news is that if 2) were implemented, every other piece of the
code can handle addition of structures dynamically, and it would be
trivial to automate updating existing profiles to handle the new
structures. it would be a simple python script that updated every
*_vtype.py file in plugins/overlays/Linux with the new structure
information.
So does anyone have a better idea than 2) or does anyone know how we
can make 2) work well? I know there are a million C file parsers out
there already, so we would just need to be able to get one to output
structures & members in some reasonable way that we can then format
into how gen_vtypes.py expects.
6:17 p.m.
New subject: Adding kernel version / distro generic Linux support to Volatility (long email & code)
Hi Andrew,
On Fri, Sep 24, 2010 at 3:02 PM, Andrew Case <atcuno(a)gmail.com> wrote:
The problem is that in the current implementation, the
file of
structure offsets that is used to make the profiles/vtypes is only the
offsets that the LKM is coded to gather. This is a problem because
every time we need to access a new structure type for some new
functionality, the LKM would need to be run again on all the target
systems in order to get the offsets that the new code requires. This
isn't something we can really deal with and we need a better solution.
Yeah it would be hard to support all the kernel versions.
2) writing something that can scan the kernels files
(not just the
header files, but all files) for a particular kernel tree and all it
would need to do is get the name of every structure and all the
members. This would also be nice b/c then instead of having to run an
LKM to get offsets, you would just have to download a particular
kernel's source package (all distros supply these) and run the script
to get all the structure names and members list. The rest of the code
could then stay how it is and we can support any access to any
structure type.
So the issue is that depending on config defines the structs would
change. The real problem is how do we get the struct offsets for the
current kernel and that really depends on the config files. One way we
can do this is to generate a vtype definition through scanning the
header files and actually vary the struct offsets based on config
options.
This is a little difficult because not only do we need to know the
config options but also how the compiler rounds the offsets for
alignments etc.
Probably a better way is to write a program which:
1) Parses the header files from the kernel headers.
2) Generates the c code for the LKM, compiles and inserts it.
3) Grabs the offsets from the LKM and generates the vtypes
Then we could extract all the offsets of all the members from the header files.
The good news is that if 2) were implemented, every
other piece of the
code can handle addition of structures dynamically, and it would be
trivial to automate updating existing profiles to handle the new
structures. it would be a simple python script that updated every
*_vtype.py file in plugins/overlays/Linux with the new structure
information.
Probably what we need to do also when we parse the header files is
document which config options influence the particular struct. We then
need to characterise profiles not by kernel versions but by config
options. So if we need a new struct we can just add it to the same
profile which is actually affected by it.
So does anyone have a better idea than 2) or does
anyone know how we
can make 2) work well? I know there are a million C file parsers out
there already, so we would just need to be able to get one to output
structures & members in some reasonable way that we can then format
into how gen_vtypes.py expects.
I dont think 2 can work without at least compiling the code because it
depends on compiler alignments etc.
Michael.
6:24 p.m.
New subject: Adding kernel version / distro generic Linux support to Volatility (long email & code)
On Fri, Sep 24, 2010 at 5:17 PM, Michael Cohen <scudette(a)gmail.com> wrote:
This is a little difficult because not only do we need to know the
config options but also how the compiler rounds the offsets for
alignments etc.
Probably a better way is to write a program which:
1) Parses the header files from the kernel headers.
2) Generates the c code for the LKM, compiles and inserts it.
3) Grabs the offsets from the LKM and generates the vtypes
Yes, this will work perfect....
So in order to accomplish this we need code that can parse a kernel
tree (including the .c files) and give us every structure and every
name. As you said, we can then generate the LKM and load it. Also, we
don't have to care about .config options then b/c we will be compiling
against the same headers & .config as the kernel was built.
Basically it boils down to something that can parse a kernel tree and
give us all structures and members. Once that is done, I can trivially
update the existing stuff to support it. Also the profiles don't have
to know about .config options, it can all be handled cleanly in the
code. I can explain this on IRC if you want, but I have already
thought the .config stuff out.
5118
days inactive
5118
days old
vol-dev@lists.volatilityfoundation.org
3 comments
2 participants
participants (2)
-
Andrew Case -
Michael Cohen