Hey Nir,
Michael, if you'll load the vmss AS with the -d flag it'll print the regions specified in the vmss file.
it might help you to check if the address really exist in the file. I would bet there's more than a single region so the file size is not indicative.i'll try running a few checks on my part to make sure it is unrelated to my patch.if you wish to investigate it further you could quite easily use the VMSSParser directly, but you might rather leave it to me :)I'll also hurry with the zread function so you could make more tests.hopefully I'll write it tomorrow, since it's 2AM over here and I've already made some promises I should keep for today :)help with the object system will be much appreciated.btw,a list of the features/methods required of an AS could have been useful, but i don't how many AS commits you receive so it might be unnecessary.I won't mind writing one if reviewed properly. an "example" AS could also do the trickCheers,- NirOn Sat, Jul 7, 2012 at 2:02 AM, Michael Hale Ligh <michael.hale@gmail.com> wrote:
Hey Nir,We can definitely help out with integrating your code with the object system. I was just in the process of testing it for the first time this afternoon.Here are a few details (I'll copy them to the issue tracker in a sec). Basically I started with an ESX 4.1.0 and grabbed the following:* vmsn from xpsp2 x86 256 MB* vmsn from win7 sp1 x86 512 MB* vmss from server 2008 sp1 x64 4 GBYour AS with the latest 2.1 alpha (so about r1983) worked fine for the xp vmsn. (by fine I mean pslist worked, but other plugins may not work properly due to what scudette said about zread)=================================================VMSN - XPSP3 x86 @ 256 MB RAM$ python vol.py -d -f Andrew-Snapshot6.vmsn pslistVolatile Systems Volatility Framework 2.1_alpha[snip]DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.vmware.VMWareSnapshotFile'>DEBUG : volatility.plugins.addrspaces.vmware: RegionCount: 1DEBUG : volatility.plugins.addrspaces.vmware: Virtual Address: 0, Physical Address: 0, Size: 10000000DEBUG : volatility.plugins.addrspaces.vmware: dtb: 39000DEBUG : volatility.utils : Succeeded instantiating <volatility.plugins.addrspaces.vmware.VMWareSnapshotFile object at 0x10363a890>Offset(V) Name PID PPID Thds Hnds Sess Wow64 Start Exit---------- -------------------- ------ ------ ------ -------- ------ ------ -------------------- --------------------0x81bcc830 System 4 0 52 477 ------ 00x8194a020 smss.exe 364 4 3 21 ------ 0 2012-01-25 20:44:200x81954020 csrss.exe 616 364 10 345 0 0 2012-01-25 20:44:200x81951128 winlogon.exe 640 364 16 495 0 0 2012-01-25 20:44:200x81a897a8 services.exe 684 640 15 272 0 0 2012-01-25 20:44:20[snip]It also worked fine for the win7 vmsn:=================================================VMSN - Windows 7 SP0 x86 @ 512 MB RAM$ python vol.py -d -f Abraham-Snapshot2.vmsn --profile=Win7SP0x86 pslistVolatile Systems Volatility Framework 2.1_alpha[snip]DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.vmware.VMWareSnapshotFile'>DEBUG : volatility.plugins.addrspaces.vmware: RegionCount: 1DEBUG : volatility.plugins.addrspaces.vmware: Virtual Address: 0, Physical Address: 0, Size: 20000000DEBUG : volatility.plugins.addrspaces.vmware: dtb: 185000DEBUG : volatility.utils : Succeeded instantiating <volatility.plugins.addrspaces.vmware.VMWareSnapshotFile object at 0x102a1a750>Offset(V) Name PID PPID Thds Hnds Sess Wow64 Start Exit---------- -------------------- ------ ------ ------ -------- ------ ------ -------------------- --------------------0x83f2f730 System 4 0 93 494 ------ 0 2012-03-15 15:04:120x84f32c48 smss.exe 252 4 2 29 ------ 0 2012-03-15 15:04:120x85708d40 csrss.exe 364 356 9 386 0 0 2012-03-15 15:04:480x82050030 wininit.exe 400 356 3 75 0 0 2012-03-15 15:04:480x856d9370 csrss.exe 408 392 7 201 1 0 2012-03-15 15:04:480x8207f030 services.exe 468 400 8 198 0 0 2012-03-15 15:04:490x8208e030 lsass.exe 476 400 8 711 0 0 2012-03-15 15:04:49[snip]I then reproduced the same thing Jesse is seeing on the server 2008 x64 w/ 4 GB:=================================================VMSS - Server 2008 SP1 x64 @ 4 GB RAM$ python vol.py -d -f Win2008SP1x64-9de64630.vmss --profile=Win2008SP1x64 pslistVolatile Systems Volatility Framework 2.1_alpha[snip]DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.hibernate.WindowsHiberFileSpace32'>DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace64'>DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace32'>DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.vmware.VMWareSnapshotFile'>DEBUG : volatility.plugins.addrspaces.vmware: Read region count from file: 2DEBUG : volatility.plugins.addrspaces.vmware: RegionCount: 2DEBUG : volatility.plugins.addrspaces.vmware: Virtual Address: 0, Physical Address: 0, Size: C0000000Virtual Address: 100000000, Physical Address: C0000000, Size: 40000000DEBUG : volatility.plugins.addrspaces.vmware: dtb: 124000DEBUG : volatility.utils : Succeeded instantiating <volatility.plugins.addrspaces.vmware.VMWareSnapshotFile object at 0x102a1a750>DEBUG : volatility.utils : Voting roundDEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.hibernate.WindowsHiberFileSpace32'>DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace64'>DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace32'>DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.vmware.VMWareSnapshotFile'>DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.amd64.AMD64PagedMemory'>DEBUG : volatility.utils : Succeeded instantiating <volatility.plugins.addrspaces.amd64.AMD64PagedMemory object at 0x1036c91d0>DEBUG : volatility.utils : Voting roundDEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.hibernate.WindowsHiberFileSpace32'>DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace64'>DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace32'>DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.vmware.VMWareSnapshotFile'>DEBUG : volatility.utils : Failed instantiating (exception): unpack requires a string argument of length 4DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.amd64.AMD64PagedMemory'>DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.intel.JKIA32PagedMemory'>DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.legacyintel.IA32PagedMemoryPae'>DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.intel.JKIA32PagedMemoryPae'>DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.legacyintel.IA32PagedMemory'>DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.standard.FileAddressSpace'>Offset(V) Name PID PPID Thds Hnds Sess Wow64 Start Exit------------------ -------------------- ------ ------ ------ -------- ------ ------ -------------------- --------------------0xfffffa8003ca8950 System 4 0 104 496 ------ 0 2012-03-02 07:16:23> /Users/Michael/volatility_pe_exceptions/volatility/plugins/overlays/windows/windows.py(262)windows_to_unix_time()-> unix_time = windows_time / 10000000(Pdb) windows_time<NoneObject: Unable to read 8 bytes from 18446738026473744904>You can see the error occurred because windows_time at 0xfffffa8004a86a08L (hex value of the decimal offset above) could not be fetched from the vmss file. Since it appears the System process is able to be found, we should be able to break into a volshell (which uses the System process AS by default) and try some checks:$ python vol.py -d -f Win2008SP1x64-9de64630.vmss --profile=Win2008SP1x64 volshellCurrent context: process System, pid=4, ppid=0 DTB=0x124000Welcome to volshell! Current memory image is:file:///Users/Michael/Downloads/Win2008SP1x64-9de64630.vmssTo get help, type 'hh()'>>> self.addrspace.is_valid_address(0xfffffa8004a86a08L)True>>> self.addrspace.vtop(0xfffffa8004a86a08L)5354580488L>>> dd(0xfffffa8004a86a08L)Memory unreadable at fffffa8004a86a08So the AS thinks the virtual address is valid and is able to vtop, but then when you try to read (dd command) it fails. The first thing that catches my eye is the physical address is reportedly 5354580488L, which is much larger than the size of the file we have:$ ls -al Win2008SP1x64-9de64630.vmss-rw-r--r--@ 1 Michael staff 4300360567 Jul 6 16:01 /Users/Michael/Downloads/Win2008SP1x64-9de64630.vmssThis is the same thing we saw recently in the issue "vtop and 5GB 64bit memory dump problem" [1]. That too was an issue of vmware memory files (a vmem in this case unless Sebastien changed the extension). Same symptom, but with the AMD64 AS - it reported vtop as being a physical address much bigger than the file.We're still looking into it some things, and although your AS could use a little work to conform its style with the other AS's, I'm not sure its the cause of the problem we're seeing here (unless the AMD64 AS has the same problem).Stay tuned.... ;-)MHLOn Fri, Jul 6, 2012 at 5:58 PM, nir izraeli <nirizr@gmail.com> wrote:hi Michael,would you mind to also post your comments at the tracking system?it'll be a lot easier for me to keep track of it. hoping I'm not stepping on your tows.about the zread() - i didn't implement it, I got confused with a few old AS classes that had unnecessary methods and probably also removed the zread() by mistake.I hope to fix these in a couple of days and resubmit an updated version.the only issue i have trouble with is the conversion to the internal object system.I tried using it a couple of times but had trouble with it.it would also duplicate efforts for writing modifications to the vmss parsing code.since it does seem to be easy to write structures using Volatility's framework,would you mind taking care of it yourselves?I could add a textual documentation if you'd rather, since i'll write one anyway.although if it's important i could give it another try...
Thanks,NirOn Fri, Jul 6, 2012 at 11:49 PM, Michael Cohen <scudette@gmail.com> wrote:
It looks to me like the address space is not implementing zread()
properly (or at all). Can you please make sure that you are
implementing zread() in such a way that when you read outside a valid
or mapped region you will receive a null padded buffer rather than
None?
Some more comments about the address space VMWareSnapshotFile:
- Please do not use inner classes. There is no need to have a class
defined in such a way - just place the class at the module level.
- Minor style issues - long lines should be wrapped at 80 chars,
commented out lines should be removed.
- Do no use double underscore member variable names (they mean
something specific e.g. self.__hasseek).
- It would also be nicer if we used the volatility object system
rather than struct module directly for parsing these things - it would
make the file formats more readable and simplify the code a lot.
Thanks
Michael.
On 6 July 2012 16:03, Jesse Bowling <jessebowling@gmail.com> wrote:
> Disclaimer:
>
> So I took Nir's files, and dropped them into my plugins folder...I did not
> see any new plugins using vol.py -h, and when I tried to do an imageinfo I
> got:
>
> /usr/local/src/volatility-read-only-may-01/vol.py -f myimage.vmss imageinfo
>
> Volatile Systems Volatility Framework 2.1_alpha
> Determining profile based on KDBG search...
>
> Traceback (most recent call last):
> File "/usr/local/src/volatility-read-only-may-01/vol.py", line 173, in
> <module>
> main()
> File "/usr/local/src/volatility-read-only-may-01/vol.py", line 164, in
> main
> command.execute()
> File "/usr/local/src/volatility-read-only-may-01/volatility/commands.py",
> line 101, in execute
> func(outfd, data)
> File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/imageinfo.py",
> line 34, in render_text
> for k, v in data:
> File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/imageinfo.py",
> line 44, in calculate
> suglist = [ s for s, _, _ in kdbg.KDBGScan.calculate(self)]
> File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/kdbgscan.py",
> line 119, in calculate
> for offset in scanner.scan(aspace):
> File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/kdbgscan.py",
> line 83, in scan
> for offset in scan.BaseScanner.scan(self, address_space, offset,
> maxlen):
> File "/usr/local/src/volatility-read-only-may-01/volatility/scan.py", line
> 136, in scan
> skip = max(skip, s.skip(data, i))
> File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/common.py",
> line 49, in skip
> nextval = data.index(self.tag, offset + 1)
> AttributeError: 'NoneType' object has no attribute 'index'
>
> So:
>
> # /usr/local/src/volatility-read-only-may-01/vol.py -f myimage.vmss psscan
>
> Volatile Systems Volatility Framework 2.1_alpha
> Offset(P) Name PID PPID PDB Time created
> Time exited
> ---------- ---------------- ------ ------ ----------
> ------------------------ ------------------------
> No suitable address space mapping found
> Tried to open image as:
> WindowsHiberFileSpace32: No base Address Space
> VMWareSnapshotFile: No base Address Space
> WindowsCrashDumpSpace32: No base Address Space
> AMD64PagedMemory: No base Address Space
> JKIA32PagedMemory: No base Address Space
> JKIA32PagedMemoryPae: No base Address Space
> IA32PagedMemoryPae: Module disabled
> IA32PagedMemory: Module disabled
> WindowsHiberFileSpace32: No xpress signature found
> WindowsHiberFileSpace32: No xpress signature found
> VMWareSnapshotFile: ('Header signature invalid', 4026597203)
> WindowsCrashDumpSpace32: Header signature invalid
> AMD64PagedMemory: Incompatible profile WinXPSP2x86 selected
> JKIA32PagedMemory: Failed valid Address Space check
> JKIA32PagedMemoryPae: Failed valid Address Space check
> IA32PagedMemoryPae: Module disabled
> IA32PagedMemory: Module disabled
> FileAddressSpace: Must be first Address Space
>
> At least it doesn't crash. So now:
>
> # /usr/local/src/volatility-read-only-may-01/vol.py -f myimage.vmss
> --profile=Win2008R2SP1x64 psscan
>
> Volatile Systems Volatility Framework 2.1_alpha
> Offset(P) Name PID PPID PDB Time created
> Time exited
> ---------- ---------------- ------ ------ ----------
> ------------------------ ------------------------
> Traceback (most recent call last):
> File "/usr/local/src/volatility-read-only-may-01/vol.py", line 173, in
> <module>
> main()
> File "/usr/local/src/volatility-read-only-may-01/vol.py", line 164, in
> main
> command.execute()
> File "/usr/local/src/volatility-read-only-may-01/volatility/commands.py",
> line 101, in execute
> func(outfd, data)
> File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/filescan.py",
> line 415, in render_text
> for eprocess in data:
> File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/filescan.py",
> line 405, in calculate
> for offset in PoolScanProcess().scan(address_space):
> File "/usr/local/src/volatility-read-only-may-01/volatility/scan.py", line
> 218, in scan
> for i in BaseScanner.scan(self, address_space, offset, maxlen):
> File "/usr/local/src/volatility-read-only-may-01/volatility/scan.py", line
> 136, in scan
> skip = max(skip, s.skip(data, i))
> File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/common.py",
> line 49, in skip
> nextval = data.index(self.tag, offset + 1)
> AttributeError: 'NoneType' object has no attribute 'index'
>
> # /usr/local/src/volatility-read-only-may-01/vol.py -f myimage.vmss
> --profile=Win2008R2SP1x64 --dtb=0x187000 psscan
>
> Volatile Systems Volatility Framework 2.1_alpha
> Offset(P) Name PID PPID PDB Time created
> Time exited
> ---------- ---------------- ------ ------ ----------
> ------------------------ ------------------------
> Traceback (most recent call last):
> File "/usr/local/src/volatility-read-only-may-01/vol.py", line 173, in
> <module>
> main()
> File "/usr/local/src/volatility-read-only-may-01/vol.py", line 164, in
> main
> command.execute()
> File "/usr/local/src/volatility-read-only-may-01/volatility/commands.py",
> line 101, in execute
> func(outfd, data)
> File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/filescan.py",
> line 415, in render_text
> for eprocess in data:
> File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/filescan.py",
> line 405, in calculate
> for offset in PoolScanProcess().scan(address_space):
> File "/usr/local/src/volatility-read-only-may-01/volatility/scan.py", line
> 218, in scan
> for i in BaseScanner.scan(self, address_space, offset, maxlen):
> File "/usr/local/src/volatility-read-only-may-01/volatility/scan.py", line
> 136, in scan
> skip = max(skip, s.skip(data, i))
> File
> "/usr/local/src/volatility-read-only-may-01/volatility/plugins/common.py",
> line 49, in skip
> nextval = data.index(self.tag, offset + 1)
> AttributeError: 'NoneType' object has no attribute 'index'
>
> I have limited testing time the next couple weeks, so will look to see if I
> can share this with someone like SA in the meantime...
>
> Cheers,
>
> Jesse
>
>
> On Fri, Jul 6, 2012 at 7:21 AM, nir izraeli <nirizr@gmail.com> wrote:
>>
>> I assume you need it for something other than test my patch,
>> I can send parts of the vmss of the machine I already noticed more than
>> one region.
>> could you use that to gather the info you need?
>>
>> btw, I'm also using vmware converter standalone pretty often, it might
>> also be related
>>
>>
>> On Fri, Jul 6, 2012 at 5:31 AM, AAron Walters <awalters@4tphi.net> wrote:
>>>
>>>
>>> Nir,
>>>
>>>
>>>> AAron - actually it was quite rare, but the first vmss I used to test
>>>> the patch
>>>> had two or three, which made my patch break when i first tested it on
>>>> other
>>>> VMs.
>>>> I could try to pinpoint it, but i guess it would be easier for me to
>>>> reverse
>>>> the vmware code than try it manually :)
>>>> A thing to note is that that vmss also had two virtual CPUs, which might
>>>> have
>>>> caused having more than one region. it also had ~4G of RAM. most of the
>>>> other
>>>> VMs i used only had about 512M.
>>>> did you try to run it on other vmss files that resemble the one i
>>>> described?
>>>
>>>
>>> Interesting. I have never seen a vmss with multiple regions. If you
>>> happen to come across one again, please let me know. I'd be interested in
>>> what conditions or what product leads to more than one region.
>>>
>>> Thanks,
>>>
>>> AW
>>
>>
>
>
>
> --
> Jesse Bowling
>
>
>
> _______________________________________________
> Vol-users mailing list
> Vol-users@volatilityfoundation.org
> http://lists.volatilityfoundation.org/mailman/listinfo/vol-users
>
_______________________________________________
Vol-users mailing list
Vol-users@volatilityfoundation.org
http://lists.volatilityfoundation.org/mailman/listinfo/vol-users