Hey Nir, 

We can definitely help out with integrating your code with the object system. I was just in the process of testing it for the first time this afternoon. 

Here are a few details (I'll copy them to the issue tracker in a sec). Basically I started with an ESX 4.1.0 and grabbed the following: 

* vmsn from xpsp2 x86 256 MB
* vmsn from win7 sp1 x86 512 MB
* vmss from server 2008 sp1 x64 4 GB 

Your AS with the latest 2.1 alpha (so about r1983) worked fine for the xp vmsn. (by fine I mean pslist worked, but other plugins may not work properly due to what scudette said about zread)

=================================================
VMSN - XPSP3 x86 @ 256 MB RAM

$ python vol.py -d -f Andrew-Snapshot6.vmsn pslist
Volatile Systems Volatility Framework 2.1_alpha
[snip]
DEBUG   : volatility.utils    : Trying <class 'volatility.plugins.addrspaces.vmware.VMWareSnapshotFile'> 
DEBUG   : volatility.plugins.addrspaces.vmware: RegionCount: 1
DEBUG   : volatility.plugins.addrspaces.vmware: Virtual Address:          0, Physical Address:        0, Size: 10000000
DEBUG   : volatility.plugins.addrspaces.vmware: dtb: 39000
DEBUG   : volatility.utils    : Succeeded instantiating <volatility.plugins.addrspaces.vmware.VMWareSnapshotFile object at 0x10363a890>
Offset(V)  Name                    PID   PPID   Thds     Hnds   Sess  Wow64 Start                Exit                
---------- -------------------- ------ ------ ------ -------- ------ ------ -------------------- --------------------
0x81bcc830 System                    4      0     52      477 ------      0                                          
0x8194a020 smss.exe                364      4      3       21 ------      0 2012-01-25 20:44:20                      
0x81954020 csrss.exe               616    364     10      345      0      0 2012-01-25 20:44:20                      
0x81951128 winlogon.exe            640    364     16      495      0      0 2012-01-25 20:44:20                      
0x81a897a8 services.exe            684    640     15      272      0      0 2012-01-25 20:44:20
[snip]

It also worked fine for the win7 vmsn:

=================================================
VMSN - Windows 7 SP0 x86 @ 512 MB RAM

$ python vol.py -d -f Abraham-Snapshot2.vmsn --profile=Win7SP0x86 pslist
Volatile Systems Volatility Framework 2.1_alpha
[snip]
DEBUG   : volatility.utils    : Trying <class 'volatility.plugins.addrspaces.vmware.VMWareSnapshotFile'> 
DEBUG   : volatility.plugins.addrspaces.vmware: RegionCount: 1
DEBUG   : volatility.plugins.addrspaces.vmware: Virtual Address:          0, Physical Address:        0, Size: 20000000
DEBUG   : volatility.plugins.addrspaces.vmware: dtb: 185000
DEBUG   : volatility.utils    : Succeeded instantiating <volatility.plugins.addrspaces.vmware.VMWareSnapshotFile object at 0x102a1a750>
Offset(V)  Name                    PID   PPID   Thds     Hnds   Sess  Wow64 Start                Exit                
---------- -------------------- ------ ------ ------ -------- ------ ------ -------------------- --------------------
0x83f2f730 System                    4      0     93      494 ------      0 2012-03-15 15:04:12                      
0x84f32c48 smss.exe                252      4      2       29 ------      0 2012-03-15 15:04:12                      
0x85708d40 csrss.exe               364    356      9      386      0      0 2012-03-15 15:04:48                      
0x82050030 wininit.exe             400    356      3       75      0      0 2012-03-15 15:04:48                      
0x856d9370 csrss.exe               408    392      7      201      1      0 2012-03-15 15:04:48                      
0x8207f030 services.exe            468    400      8      198      0      0 2012-03-15 15:04:49                      
0x8208e030 lsass.exe               476    400      8      711      0      0 2012-03-15 15:04:49 
[snip]

I then reproduced the same thing Jesse is seeing on the server 2008 x64 w/ 4 GB:

=================================================
VMSS - Server 2008 SP1 x64 @ 4 GB RAM

$ python vol.py -d -f Win2008SP1x64-9de64630.vmss --profile=Win2008SP1x64 pslist
Volatile Systems Volatility Framework 2.1_alpha
[snip]
DEBUG   : volatility.utils    : Trying <class 'volatility.plugins.addrspaces.hibernate.WindowsHiberFileSpace32'> 
DEBUG   : volatility.utils    : Trying <class 'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace64'> 
DEBUG   : volatility.utils    : Trying <class 'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace32'> 
DEBUG   : volatility.utils    : Trying <class 'volatility.plugins.addrspaces.vmware.VMWareSnapshotFile'> 
DEBUG   : volatility.plugins.addrspaces.vmware: Read region count from file: 2
DEBUG   : volatility.plugins.addrspaces.vmware: RegionCount: 2
DEBUG   : volatility.plugins.addrspaces.vmware: Virtual Address:          0, Physical Address:        0, Size: C0000000
Virtual Address:  100000000, Physical Address: C0000000, Size: 40000000
DEBUG   : volatility.plugins.addrspaces.vmware: dtb: 124000
DEBUG   : volatility.utils    : Succeeded instantiating <volatility.plugins.addrspaces.vmware.VMWareSnapshotFile object at 0x102a1a750>
DEBUG   : volatility.utils    : Voting round
DEBUG   : volatility.utils    : Trying <class 'volatility.plugins.addrspaces.hibernate.WindowsHiberFileSpace32'> 
DEBUG   : volatility.utils    : Trying <class 'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace64'> 
DEBUG   : volatility.utils    : Trying <class 'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace32'> 
DEBUG   : volatility.utils    : Trying <class 'volatility.plugins.addrspaces.vmware.VMWareSnapshotFile'> 
DEBUG   : volatility.utils    : Trying <class 'volatility.plugins.addrspaces.amd64.AMD64PagedMemory'> 
DEBUG   : volatility.utils    : Succeeded instantiating <volatility.plugins.addrspaces.amd64.AMD64PagedMemory object at 0x1036c91d0>
DEBUG   : volatility.utils    : Voting round
DEBUG   : volatility.utils    : Trying <class 'volatility.plugins.addrspaces.hibernate.WindowsHiberFileSpace32'> 
DEBUG   : volatility.utils    : Trying <class 'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace64'> 
DEBUG   : volatility.utils    : Trying <class 'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace32'> 
DEBUG   : volatility.utils    : Trying <class 'volatility.plugins.addrspaces.vmware.VMWareSnapshotFile'> 
DEBUG   : volatility.utils    : Failed instantiating (exception): unpack requires a string argument of length 4
DEBUG   : volatility.utils    : Trying <class 'volatility.plugins.addrspaces.amd64.AMD64PagedMemory'> 
DEBUG   : volatility.utils    : Trying <class 'volatility.plugins.addrspaces.intel.JKIA32PagedMemory'> 
DEBUG   : volatility.utils    : Trying <class 'volatility.plugins.addrspaces.legacyintel.IA32PagedMemoryPae'> 
DEBUG   : volatility.utils    : Trying <class 'volatility.plugins.addrspaces.intel.JKIA32PagedMemoryPae'> 
DEBUG   : volatility.utils    : Trying <class 'volatility.plugins.addrspaces.legacyintel.IA32PagedMemory'> 
DEBUG   : volatility.utils    : Trying <class 'volatility.plugins.addrspaces.standard.FileAddressSpace'> 
Offset(V)          Name                    PID   PPID   Thds     Hnds   Sess  Wow64 Start                Exit                
------------------ -------------------- ------ ------ ------ -------- ------ ------ -------------------- --------------------
0xfffffa8003ca8950 System                    4      0    104      496 ------      0 2012-03-02 07:16:23                      
> /Users/Michael/volatility_pe_exceptions/volatility/plugins/overlays/windows/windows.py(262)windows_to_unix_time()
-> unix_time = windows_time / 10000000
(Pdb) windows_time
<NoneObject: Unable to read 8 bytes from 18446738026473744904>

You can see the error occurred because windows_time at 0xfffffa8004a86a08L (hex value of the decimal offset above) could not be fetched from the vmss file. Since it appears the System process is able to be found, we should be able to break into a volshell (which uses the System process AS by default) and try some checks:

$ python vol.py -d -f Win2008SP1x64-9de64630.vmss --profile=Win2008SP1x64 volshell
Current context: process System, pid=4, ppid=0 DTB=0x124000
Welcome to volshell! Current memory image is:
file:///Users/Michael/Downloads/Win2008SP1x64-9de64630.vmss
To get help, type 'hh()'
>>> self.addrspace.is_valid_address(0xfffffa8004a86a08L)
True
>>> self.addrspace.vtop(0xfffffa8004a86a08L)
5354580488L
>>> dd(0xfffffa8004a86a08L)
Memory unreadable at fffffa8004a86a08

So the AS thinks the virtual address is valid and is able to vtop, but then when you try to read (dd command) it fails. The first thing that catches my eye is the physical address is reportedly 5354580488L, which is much larger than the size of the file we have:

$ ls -al Win2008SP1x64-9de64630.vmss
-rw-r--r--@ 1 Michael  staff  4300360567 Jul  6 16:01 /Users/Michael/Downloads/Win2008SP1x64-9de64630.vmss

This is the same thing we saw recently in the issue "vtop and 5GB 64bit memory dump problem" [1]. That too was an issue of vmware memory files (a vmem in this case unless Sebastien changed the extension). Same symptom, but with the AMD64 AS - it reported vtop as being a physical address much bigger than the file.

We're still looking into it some things, and although your AS could use a little work to conform its style with the other AS's, I'm not sure its the cause of the problem we're seeing here (unless the AMD64 AS has the same problem). 

Stay tuned.... ;-)

MHL

[1].  http://code.google.com/p/volatility/issues/detail?id=272

On Fri, Jul 6, 2012 at 5:58 PM, nir izraeli <nirizr@gmail.com> wrote:
hi Michael,
would you mind to also post your comments at the tracking system?
it'll be a lot easier for me to keep track of it. hoping I'm not stepping on your tows.

about the zread() - i didn't implement it, I got confused with a few old AS classes that had unnecessary methods and probably also removed the zread() by mistake.

I hope to fix these in a couple of days and resubmit an updated version.

the only issue i have trouble with is the conversion to the internal object system.
I tried using it a couple of times but had trouble with it.
it would also duplicate efforts for writing modifications to the vmss parsing code.
since it does seem to be easy to write structures using Volatility's framework,
would you mind taking care of it yourselves?
I could add a textual documentation if you'd rather, since i'll write one anyway. 
although if it's important i could give it another try...

Thanks,
Nir


On Fri, Jul 6, 2012 at 11:49 PM, Michael Cohen <scudette@gmail.com> wrote:
It looks to me like the address space is not implementing zread()
properly (or at all). Can you please make sure that you are
implementing zread() in such a way that when you read outside a valid
or mapped region you will receive a null padded buffer rather than
None?

Some more comments about the address space VMWareSnapshotFile:
   - Please do not use inner classes. There is no need to have a class
defined in such a way - just place the class at the module level.
   - Minor style issues - long lines should be wrapped at 80 chars,
commented out lines should be removed.
   - Do no use double underscore member variable names (they mean
something specific e.g. self.__hasseek).
   - It would also be nicer if we used the volatility object system
rather than struct module directly for parsing these things - it would
make the file formats more readable and simplify the code a lot.

Thanks
Michael.

On 6 July 2012 16:03, Jesse Bowling <jessebowling@gmail.com> wrote:
> Disclaimer:
>
> So I took Nir's files, and dropped them into my plugins folder...I did not
> see any new plugins using vol.py -h, and when I tried to do an imageinfo I
> got:
>
> /usr/local/src/volatility-read-only-may-01/vol.py -f myimage.vmss imageinfo
>
> Volatile Systems Volatility Framework 2.1_alpha
> Determining profile based on KDBG search...
>
> Traceback (most recent call last):
>   File "/usr/local/src/volatility-read-only-may-01/vol.py", line 173, in
> <module>
>     main()
>   File "/usr/local/src/volatility-read-only-may-01/vol.py", line 164, in
> main
>     command.execute()
>   File "/usr/local/src/volatility-read-only-may-01/volatility/commands.py",
> line 101, in execute
>     func(outfd, data)
>   File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/imageinfo.py",
> line 34, in render_text
>     for k, v in data:
>   File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/imageinfo.py",
> line 44, in calculate
>     suglist = [ s for s, _, _ in kdbg.KDBGScan.calculate(self)]
>   File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/kdbgscan.py",
> line 119, in calculate
>     for offset in scanner.scan(aspace):
>   File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/kdbgscan.py",
> line 83, in scan
>     for offset in scan.BaseScanner.scan(self, address_space, offset,
> maxlen):
>   File "/usr/local/src/volatility-read-only-may-01/volatility/scan.py", line
> 136, in scan
>     skip = max(skip, s.skip(data, i))
>   File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/common.py",
> line 49, in skip
>     nextval = data.index(self.tag, offset + 1)
> AttributeError: 'NoneType' object has no attribute 'index'
>
> So:
>
> # /usr/local/src/volatility-read-only-may-01/vol.py -f myimage.vmss psscan
>
> Volatile Systems Volatility Framework 2.1_alpha
>  Offset(P)  Name             PID    PPID   PDB        Time created
> Time exited
> ---------- ---------------- ------ ------ ----------
> ------------------------ ------------------------
> No suitable address space mapping found
> Tried to open image as:
>  WindowsHiberFileSpace32: No base Address Space
>  VMWareSnapshotFile: No base Address Space
>  WindowsCrashDumpSpace32: No base Address Space
>  AMD64PagedMemory: No base Address Space
>  JKIA32PagedMemory: No base Address Space
>  JKIA32PagedMemoryPae: No base Address Space
>  IA32PagedMemoryPae: Module disabled
>  IA32PagedMemory: Module disabled
>  WindowsHiberFileSpace32: No xpress signature found
>  WindowsHiberFileSpace32: No xpress signature found
>  VMWareSnapshotFile: ('Header signature invalid', 4026597203)
>  WindowsCrashDumpSpace32: Header signature invalid
>  AMD64PagedMemory: Incompatible profile WinXPSP2x86 selected
>  JKIA32PagedMemory: Failed valid Address Space check
>  JKIA32PagedMemoryPae: Failed valid Address Space check
>  IA32PagedMemoryPae: Module disabled
>  IA32PagedMemory: Module disabled
>  FileAddressSpace: Must be first Address Space
>
> At least it doesn't crash. So now:
>
> # /usr/local/src/volatility-read-only-may-01/vol.py -f myimage.vmss
> --profile=Win2008R2SP1x64 psscan
>
> Volatile Systems Volatility Framework 2.1_alpha
>  Offset(P)  Name             PID    PPID   PDB        Time created
> Time exited
> ---------- ---------------- ------ ------ ----------
> ------------------------ ------------------------
> Traceback (most recent call last):
>   File "/usr/local/src/volatility-read-only-may-01/vol.py", line 173, in
> <module>
>     main()
>   File "/usr/local/src/volatility-read-only-may-01/vol.py", line 164, in
> main
>     command.execute()
>   File "/usr/local/src/volatility-read-only-may-01/volatility/commands.py",
> line 101, in execute
>     func(outfd, data)
>   File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/filescan.py",
> line 415, in render_text
>     for eprocess in data:
>   File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/filescan.py",
> line 405, in calculate
>     for offset in PoolScanProcess().scan(address_space):
>   File "/usr/local/src/volatility-read-only-may-01/volatility/scan.py", line
> 218, in scan
>     for i in BaseScanner.scan(self, address_space, offset, maxlen):
>   File "/usr/local/src/volatility-read-only-may-01/volatility/scan.py", line
> 136, in scan
>     skip = max(skip, s.skip(data, i))
>   File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/common.py",
> line 49, in skip
>     nextval = data.index(self.tag, offset + 1)
> AttributeError: 'NoneType' object has no attribute 'index'
>
> # /usr/local/src/volatility-read-only-may-01/vol.py -f myimage.vmss
> --profile=Win2008R2SP1x64 --dtb=0x187000 psscan
>
> Volatile Systems Volatility Framework 2.1_alpha
>  Offset(P)  Name             PID    PPID   PDB        Time created
> Time exited
> ---------- ---------------- ------ ------ ----------
> ------------------------ ------------------------
> Traceback (most recent call last):
>   File "/usr/local/src/volatility-read-only-may-01/vol.py", line 173, in
> <module>
>     main()
>   File "/usr/local/src/volatility-read-only-may-01/vol.py", line 164, in
> main
>     command.execute()
>   File "/usr/local/src/volatility-read-only-may-01/volatility/commands.py",
> line 101, in execute
>     func(outfd, data)
>   File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/filescan.py",
> line 415, in render_text
>     for eprocess in data:
>   File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/filescan.py",
> line 405, in calculate
>     for offset in PoolScanProcess().scan(address_space):
>   File "/usr/local/src/volatility-read-only-may-01/volatility/scan.py", line
> 218, in scan
>     for i in BaseScanner.scan(self, address_space, offset, maxlen):
>   File "/usr/local/src/volatility-read-only-may-01/volatility/scan.py", line
> 136, in scan
>     skip = max(skip, s.skip(data, i))
>   File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/common.py",
> line 49, in skip
>     nextval = data.index(self.tag, offset + 1)
> AttributeError: 'NoneType' object has no attribute 'index'
>
> I have limited testing time the next couple weeks, so will look to see if I
> can share this with someone like SA in the meantime...
>
> Cheers,
>
> Jesse
>
>
> On Fri, Jul 6, 2012 at 7:21 AM, nir izraeli <nirizr@gmail.com> wrote:
>>
>> I assume you need it for something other than test my patch,
>> I can send parts of the vmss of the machine I already noticed more than
>> one region.
>> could you use that to gather the info you need?
>>
>> btw, I'm also using vmware converter standalone pretty often, it might
>> also be related
>>
>>
>> On Fri, Jul 6, 2012 at 5:31 AM, AAron Walters <awalters@4tphi.net> wrote:
>>>
>>>
>>> Nir,
>>>
>>>
>>>> AAron - actually it was quite rare, but the first vmss I used to test
>>>> the patch
>>>> had two or three, which made my patch break when i first tested it on
>>>> other
>>>> VMs.
>>>> I could try to pinpoint it, but i guess it would be easier for me to
>>>> reverse
>>>> the vmware code than try it manually :)
>>>> A thing to note is that that vmss also had two virtual CPUs, which might
>>>> have
>>>> caused having more than one region. it also had ~4G of RAM. most of the
>>>> other
>>>> VMs i used only had about 512M.
>>>> did you try to run it on other vmss files that resemble the one i
>>>> described?
>>>
>>>
>>> Interesting.  I have never seen a vmss with multiple regions. If you
>>> happen to come across one again, please let me know. I'd be interested in
>>> what conditions or what product leads to more than one region.
>>>
>>> Thanks,
>>>
>>> AW
>>
>>
>
>
>
> --
> Jesse Bowling
>
>
>
> _______________________________________________
> Vol-users mailing list
> Vol-users@volatilityfoundation.org
> http://lists.volatilityfoundation.org/mailman/listinfo/vol-users
>


_______________________________________________
Vol-users mailing list
Vol-users@volatilesystems.com
http://lists.volatilityfoundation.org/mailman/listinfo/vol-users