New subject: [patch] Pylint improvements for new_object

22 Sep 2009

      Hi Mike,
 Thanks for the patch. I tried to update it in the spirit of the new
object framework.
Im sure you already know what im about to say but this is to clarify
for other members of the list.

When you add a vtype overlay like

+    '_KUSER_SHARED_DATA' : [ None, { \
+    'SystemTime' : [ None, ['WinTimeStamp', {}]], \
+    'TimeZoneBias' : [ None, ['WinTimeStamp', {}]], \
+    }],

You are saying that a WinTimeStamp object should be used to decode
these fields. This overrides the standard definition in vtypes.py:

 '_KUSER_SHARED_DATA' : [ 0x338, { \
   'SystemTime' : [ 0x14, ['_KSYSTEM_TIME']], \
   'TimeZoneBias' : [ 0x20, ['_KSYSTEM_TIME']], \
   'SuiteMask' : [ 0x2d0, ['unsigned long']], \
   'NumberOfPhysicalPages' : [ 0x2e8, ['unsigned long']], \

so in your ident module when you do this:

k = object2.NewObject("_KUSER_SHARED_DATA",
win32.info.KUSER_SHARED_DATA, addr_space, profile=self.profile)

you will actually get an instance of WinTimeStamp back. Since adding
and subtracting time is quite common its worth adding a __sub__ method
to this class so you can just do:

return k.SystemTime - k.TimeZoneBias

Which is more readable and easier to understand - also note that this
should return a WinTimeStamp object - typically with the new command
reorg we try to ensure that we dont pass strings until the very end -
so as not to lose information. Note that expanding a WinTimeStamp into
a string will format it. You should try to use construct like
outfd.write("                Datetime: %s\n" % data['ImageDatetime'])

 instead of

outfd.write("                Datetime: " + data['ImageDatetime'])

The latter breaks when data is not a string (which it should not usually be)

Finally try to return NoneObject( reason) from functions rather than
None - this allows you to do stuff like:

       for task in win32.tasks.pslist(addr_space, self.profile):
           if task.Peb.CSDVersion:

instead of

+        for task in win32.tasks.pslist(addr_space, self.profile):
+            if task.Peb is not None:
+                if task.Peb.CSDVersion is not None:

Which is more error prone because you can forget to check for None at
a certain level for a timebomb bug. Admittadly I fixed a bug where
pslist() didnt do this it was my fault - but thats what we aim for.
Note that NoneObjects can be deferenced as much as you like to produce
the same NoneObject (i.e. the original fault is propagated).

This is important since in memory forensics sometime you will fail to
dereference a point which you thought was there (e.g. if its paged
out) so at least this allows the code to proceed without raising and
prevents you from having to check every single dereference for
success.

Hope this help,
Michael.

On Mon, Sep 21, 2009 at 8:34 AM, Mike Auty &lt;mike.auty(a)gmail.com&gt; wrote:
...
  Michael Cohen wrote:
  Hi Mike, 
 Hiya Michael!

  Ok I will push it to the repository. I must admit
im not a fan of svn
 so im still a bit rusty in using it. Is there any way to get you
 access to the new object branch? (this question is directed at project
 admins btw). 
 Thanks very much for committing that.  I know what you mean about svn, I
 tend to use git to track everything, and then it can just push back into
 svn when it needs too (and it'll remind you if you missed a file, just
 like hg!).  5;)

  Ahh the linux tasks is really not even integrated
at all yet. Its kind
 of neglected since noone seems to be interested in linux atm :-(. 
 I think Jon (echo6) might be interested in the Linux side of things.
 I've also got the old truecrypt plugin on look at on my todo list (way,
 way down on my todo list, but there at least), so we might be able to
 help bolster that.  It might be more worthwhile persuing the Mac OS
 X/BSD side of things though, given the distribution of machines people
 might run up against (particularly if people start looking at iPhones on
 ARM)...  5:)

  heh - see here is my svn ignorance showing again
- svn doesnt
 automatically tell you you missed adding a file as far as i can see
 (like hg does) - sorry I forgot to add that file. 
 No problem, thanks for adding it back in.  5:)

  The scan2 stuff is faster because its a more
optimised scanning
 technique - not really related to the new object framework. Basically
 I was going to convert all the basic modules to the new framework as
 well as we can. It will require a re-engineering of the module rather
 than just a search/replace though since the new framework is much more
 concise and readable - case in point is the example I gave previously
 of _CM_HIVE. 
 Thanks, I had a go at starting to convert over some of the modules.
 I've done datetime and ident for now, just to get me going.  Any chance
 you could take a look at them (part of the attached patch, since I
 touched other bits of the tree).  I'm still not fully up to speed on
 everything, so any criticisms would be greatly appreciated!  5:)

  It will be clearer quite quickly which ones we
need and which are not
 so needed - e.g. the pstree one is basically pslist with some extra
 stuff.  If you look at the memory_objects//Windows/xp_sp2.py you can
 see that the new object classes are starting to implement their own
 methods to do stuff they want - for example to iterate over an
 _EPROCESS's handles you can just call:

 for h in process.handles():
     xxx

 this basically obsoletes Volatility/forensics/win32/handles.py 
 That's very cool!  The next step I guess will be to start removing the
 older/unnecessary stuff to clear it all up.  5:)

  Thats a great idea - although better than a
single version string it
 might be better to implement separate versions for each subsystem and
 let the plugin decide if its ok to work with - e.g. new_object_version
 1.0 etc. Maybe just have a version in the command class? 
 Yep, that seems reasonable.  The command class can keep a requirements
 list of features, and the framework can check it satisfies them all.
 That will allow for newer versions and new features, and to gracefully
 deprecate then remove old versions.

  It might work this way im not sure - but the
intension is to produce
 better code, so I would suggest a better replacement for each plugin
 should be written in the spirit of the new object design. 
 Absolutely, as I say, I've started with the easy ones, but I have every
 intention to moving getting rid of vmodules (and then clearing out as
 much of the win32.* as are no longer necessary/use the old object model.

  Oh i wrote a page about it once -
 http://www.pyflag.net/cgi-bin/moin.cgi/Volatility_Object_Framework

 This is just me playing around and not any "official" way to generate
 vtypes. It should probably be added to the repo. 
 Brilliant, that's just the kind of documentation I was looking for.  It
 gives a good overview of what's going on.  Thanks!  5:)

  Developement typically occured ad hoc and people
 needed to essentially patch the framework as well as provide plugins
 which sort of defeats the whole point of plugins. One of the problems
 has been the inability to add complex vtype handling via plugins - a
 problem which is addressed in the new branch. 
 The overlay idea looks really good, but I'm still a little unclear how
 to integrate it cleanly (ie, how to have it affect just a single module,
 in case one overlays changes that break another)...

  ## Maintained for backward compatibility do not
use in new code 
 Yeah, I saw that, it was just still used all over the place, and unless
 it goes away, I think people will keep using it...  5:(

  I was a bit reluctant to completely remove old
object support - but I
 think I might need to do that to make it clear what files are going to
 be removed. 
 Yeah, I think so too, but it should make everything a lot easier to
 follow (and much easier to document!).  5:)

 Mike  5:)

Re: [Vol-dev] [patch] Pylint improvements for new_object