Analyzing large KVM/libvrt dumps
by Juerg Haefliger
Hi all,
I wrote a little tool to convert a KVM/libvirt dump to a raw memory
file (https://github.com/juergh/lqs2mem) Volatility seems to be able
to handle the resulting file just fine for small dumps but not so much
the larger they get. Specifically, things start to break when the
memory size of the VM approaches 4 GB. I double and triple checked my
code and can't find anything obviously wrong (like using a 32bit
variable for a 64bit address or pointer). I also don't think that
Volatility has a problem with larger dumps since it can handle a 8 GB
memory dump that I obtained using some other means. I'm just running
out of ideas and am looking for some help or suggestions on how to
debug this further.
In my testing with Win 2008 R2 SP1 x64 I found that (see full outputs below):
1) imageinfo and pslist return the correct output for VMs with less than 3588 MB
2) pslist only returns a single task (System) for VMs larger than 3587 MB
3) imageinfo shows only 1 processor (when there are actually two) for
VMs larger than 3712 MB (give or take)
Any help is greatly appreciated.
Thanks
...Juerg
VM memory size: 3584 MB:
Determining profile based on KDBG search...
Suggested Profile(s) : Win2008R2SP0x64, Win7SP1x64,
Win7SP0x64, Win2008R2SP1x64
AS Layer1 : AMD64PagedMemory (Kernel AS)
AS Layer2 : FileAddressSpace
(/var/lib/libvirt/qemu/save/win-3584.ram)
PAE type : PAE
DTB : 0x187000L
KDBG : 0xf800017fb0a0
Number of Processors : 2
Image Type (Service Pack) : 1
KPCR for CPU 0 : 0xfffff800017fcd00L
KPCR for CPU 1 : 0xfffff880009b8000L
KUSER_SHARED_DATA : 0xfffff78000000000L
Image date and time : 2013-07-16 12:24:50 UTC+0000
Image local date and time : 2013-07-16 12:24:50 +0000
Offset(V) Name PID PPID Thds Hnds
Sess Wow64 Start Exit
------------------ -------------------- ------ ------ ------ --------
------ ------ ------------------------------
------------------------------
0xfffffa8002a7cb30 System 4 0 70 396
------ 0 2013-07-16 12:24:33 UTC+0000
0xfffffa80030f09d0 smss.exe 220 4 4 31
------ 0 2013-07-16 12:24:33 UTC+0000
0xfffffa80034574d0 csrss.exe 300 292 9 339
0 0 2013-07-16 12:24:34 UTC+0000
0xfffffa8003465b30 wininit.exe 352 292 7 93
0 0 2013-07-16 12:24:34 UTC+0000
0xfffffa8003469b30 csrss.exe 368 344 8 76
1 0 2013-07-16 12:24:34 UTC+0000
0xfffffa800349c280 winlogon.exe 412 344 5 83
1 0 2013-07-16 12:24:34 UTC+0000
0xfffffa80034a7160 services.exe 448 352 17 215
0 0 2013-07-16 12:24:34 UTC+0000
0xfffffa80034b4b30 lsass.exe 464 352 9 458
0 0 2013-07-16 12:24:34 UTC+0000
0xfffffa80034b64f0 lsm.exe 472 352 12 194
0 0 2013-07-16 12:24:34 UTC+0000
0xfffffa800350cb30 svchost.exe 584 448 17 355
0 0 2013-07-16 12:24:34 UTC+0000
0xfffffa8003522060 svchost.exe 664 448 13 221
0 0 2013-07-16 12:24:34 UTC+0000
0xfffffa8003547060 svchost.exe 724 448 16 312
0 0 2013-07-16 12:24:34 UTC+0000
0xfffffa8003552b30 LogonUI.exe 744 412 8 157
1 0 2013-07-16 12:24:34 UTC+0000
0xfffffa8003572b30 svchost.exe 812 448 43 782
0 0 2013-07-16 12:24:34 UTC+0000
0xfffffa8003594b30 svchost.exe 856 448 14 234
0 0 2013-07-16 12:24:34 UTC+0000
0xfffffa800359b9b0 svchost.exe 900 448 8 128
0 0 2013-07-16 12:24:34 UTC+0000
0xfffffa80035b3060 svchost.exe 940 448 19 361
0 0 2013-07-16 12:24:34 UTC+0000
0xfffffa80035fcb30 svchost.exe 372 448 16 259
0 0 2013-07-16 12:24:35 UTC+0000
0xfffffa80035f6b30 spoolsv.exe 1048 448 8 89
0 0 2013-07-16 12:24:35 UTC+0000
0xfffffa8003679650 blnsvr.exe 1076 448 7 100
0 0 2013-07-16 12:24:35 UTC+0000
0xfffffa80035e5450 svchost.exe 1116 448 4 50
0 0 2013-07-16 12:24:35 UTC+0000
0xfffffa8003732b30 WmiPrvSE.exe 1364 584 15 294
0 0 2013-07-16 12:24:35 UTC+0000
0xfffffa8003767250 svchost.exe 1484 448 12 241
0 0 2013-07-16 12:24:35 UTC+0000
0xfffffa80037df620 WmiApSrv.exe 1684 448 7 112
0 0 2013-07-16 12:24:36 UTC+0000
0xfffffa80037a56c0 WmiPrvSE.exe 1716 584 7 105
0 0 2013-07-16 12:24:36 UTC+0000
0xfffffa8003763270 WmiPrvSE.exe 1764 584 7 175
0 0 2013-07-16 12:24:38 UTC+0000
VM memory size: 3588 MB
Determining profile based on KDBG search...
Suggested Profile(s) : Win2008R2SP0x64, Win7SP1x64,
Win7SP0x64, Win2008R2SP1x64
AS Layer1 : AMD64PagedMemory (Kernel AS)
AS Layer2 : FileAddressSpace
(/var/lib/libvirt/qemu/save/win-3588.ram)
PAE type : PAE
DTB : 0x187000L
KDBG : 0xf8000180e0a0
Number of Processors : 2
Image Type (Service Pack) : 1
KPCR for CPU 0 : 0xfffff8000180fd00L
KPCR for CPU 1 : 0xfffff880009b8000L
KUSER_SHARED_DATA : 0xfffff78000000000L
Image date and time : 2013-07-16 12:50:59 UTC+0000
Image local date and time : 2013-07-16 12:50:59 +0000
Offset(V) Name PID PPID Thds Hnds
Sess Wow64 Start Exit
------------------ -------------------- ------ ------ ------ --------
------ ------ ------------------------------
------------------------------
0xfffffa800308d9e0 System 4 0 68 275
------ 0 2013-07-16 12:50:55 UTC+0000
VM memory size: 3840 MB
Determining profile based on KDBG search...
Suggested Profile(s) : Win2008R2SP0x64, Win7SP1x64,
Win7SP0x64, Win2008R2SP1x64
AS Layer1 : AMD64PagedMemory (Kernel AS)
AS Layer2 : FileAddressSpace
(/var/lib/libvirt/qemu/save/win-3840.ram)
PAE type : PAE
DTB : 0x187000L
KDBG : 0xf800018400a0
Number of Processors : 1
Image Type (Service Pack) : 1
KPCR for CPU 0 : 0xfffff80001841d00L
KUSER_SHARED_DATA : 0xfffff78000000000L
Image date and time : 2013-07-16 12:28:55 UTC+0000
Image local date and time : 2013-07-16 12:28:55 +0000
Offset(V) Name PID PPID Thds Hnds
Sess Wow64 Start Exit
------------------ -------------------- ------ ------ ------ --------
------ ------ ------------------------------
------------------------------
0xfffffa80033849e0 System 4 0 72 --------
------ 0 2013-07-16 12:28:47 UTC+0000
12 years
- 5
- 8
Extracting Memory Mapped/Cached Files
by AAron Walters
vol-users,
During last year's OMFW, I gave a presentation on a new Volatility plugin
called dumpfiles[1]. This plugin automates the process of extracting both
memory mapped and cached files. While we have distributed early versions
of the plugin in the Volatility training classes, we are in the final
stages of testing for its inclusion in the upcoming 2.3 release. If you
have some cycles and interest in helping us test, please send me a note
off-list.
Thanks,
AW
PS: Special thanks to Ikelos, MHL, Gleeda, attc, and Carl Pulley for their
help with earlier versions!
[1] http://volatility-labs.blogspot.com/2012/10/movp-44-cache-rules-everything-…
12 years
- 1
- 0
Amsterdam class sold out, next public training is in Reston, VA in November
by Andrew Case
We are writing to announce that our upcoming Amsterdam class is
completely sold out and that our next public training will be in
Reston, VA in November. We have a blog post on the Reston training
here:
http://volatility-labs.blogspot.com/2013/06/memory-forensics-training-resto…
This will be our last public training of 2013 and we are actively
planning 2014 trainings. These will be announced when they are ready.
Our last Reston class sold out a full month in advance so please
contact us ASAP if you would like to attend.
12 years
- 1
- 0
Reminder: OMFW 2013
by AAron Walters
Volatility Community,
If you are planning to attend the Open Memory Forensics Workshop 2013, I
would suggest registering soon. We only have a limited number of seats
remaining:
https://www.volatilityfoundation.org/default/omfw
PS: The only way to reserve a seat is by emailing:
info [at] volatilesystems [dot] com
Thanks,
AAron Walters
The Volatility Foundation
12 years
- 1
- 0
Re: [Vol-users] Analyzing large KVM/libvrt dumps
by Sebastien Bourdon-Richard
Juerg,
*Or are you saying that I need to shift everything resulting in a file that
is bigger than the actual physical RAM size of the VM?*
Yes. Physical address space is always bigger than physical RAM because it
contains device memory (
http://blogs.technet.com/blogfiles/markrussinovich/WindowsLiveWriter/Pushin…
).
*In my testing with Win 2008 R2 SP1 x64 I found that (see full outputs
below):*
*1) imageinfo and pslist return the correct output for VMs with less than
3588 MB
2) pslist only returns a single task (System) for VMs larger than 3587 MB*
I think important structure used by pslist are usually map over 0x100000000
on Windows 7/2008 with more that 3.5GB (approximately, depending on the
hardware installed).
During my (limited) tests, I was not able to run pslist on those OS without
the proper padding of my vmem files:
https://volatility.googlecode.com/issues/attachment?aid=2720017001&name=Vme…
Sebastien
On Wed, Aug 7, 2013 at 12:06 PM, Juerg Haefliger <juergh(a)gmail.com> wrote:
> Hi Sebastien,
>
>
> > Hello Juerg,
> >
> > Your issues seems to be similar to the one I had with VmWare
> Workstation. To
> > solve the problem, I have wrote a vmem address space that use vmss
> metadata
> > to pad the hardware range:
> >
> > https://code.google.com/p/volatility/issues/detail?id=272#c17
>
> I read through that email chain but don't claim to understand it all.
>
>
> > Maybe you need to do something similar with KVM.
> >
> > It depends on the hardware installed on your PC, but most of the time
> (on my
> > PCs), the range to pad was between 0xC0000000 - 0x100000000
>
> Hmm... The KVM file contains page addresses that I use to seek in the
> output file. If there are no pages for the 0xc000000 - 0x10000000
> range than that part of the output file will just contain garbage. Or
> are you saying that I need to shift everything resulting in a file
> that is bigger than the actual physical RAM size of the VM?
>
> ...Juerg
>
>
> > Sebastien
> >
> > On Wed, Aug 7, 2013 at 7:20 AM, Juerg Haefliger <juergh(a)gmail.com>
> wrote:
> >>
> >> Hi all,
> >>
> >> I wrote a little tool to convert a KVM/libvirt dump to a raw memory
> >> file (https://github.com/juergh/lqs2mem) Volatility seems to be able
> >> to handle the resulting file just fine for small dumps but not so much
> >> the larger they get. Specifically, things start to break when the
> >> memory size of the VM approaches 4 GB. I double and triple checked my
> >> code and can't find anything obviously wrong (like using a 32bit
> >> variable for a 64bit address or pointer). I also don't think that
> >> Volatility has a problem with larger dumps since it can handle a 8 GB
> >> memory dump that I obtained using some other means. I'm just running
> >> out of ideas and am looking for some help or suggestions on how to
> >> debug this further.
> >>
> >> In my testing with Win 2008 R2 SP1 x64 I found that (see full outputs
> >> below):
> >>
> >> 1) imageinfo and pslist return the correct output for VMs with less than
> >> 3588 MB
> >> 2) pslist only returns a single task (System) for VMs larger than 3587
> MB
> >> 3) imageinfo shows only 1 processor (when there are actually two) for
> >> VMs larger than 3712 MB (give or take)
> >>
> >> Any help is greatly appreciated.
> >>
> >> Thanks
> >> ...Juerg
> >>
> >>
> >>
> >>
> >> VM memory size: 3584 MB:
> >>
> >> Determining profile based on KDBG search...
> >>
> >> Suggested Profile(s) : Win2008R2SP0x64, Win7SP1x64,
> >> Win7SP0x64, Win2008R2SP1x64
> >> AS Layer1 : AMD64PagedMemory (Kernel AS)
> >> AS Layer2 : FileAddressSpace
> >> (/var/lib/libvirt/qemu/save/win-3584.ram)
> >> PAE type : PAE
> >> DTB : 0x187000L
> >> KDBG : 0xf800017fb0a0
> >> Number of Processors : 2
> >> Image Type (Service Pack) : 1
> >> KPCR for CPU 0 : 0xfffff800017fcd00L
> >> KPCR for CPU 1 : 0xfffff880009b8000L
> >> KUSER_SHARED_DATA : 0xfffff78000000000L
> >> Image date and time : 2013-07-16 12:24:50 UTC+0000
> >> Image local date and time : 2013-07-16 12:24:50 +0000
> >>
> >> Offset(V) Name PID PPID Thds Hnds
> >> Sess Wow64 Start Exit
> >> ------------------ -------------------- ------ ------ ------ --------
> >> ------ ------ ------------------------------
> >> ------------------------------
> >> 0xfffffa8002a7cb30 System 4 0 70 396
> >> ------ 0 2013-07-16 12:24:33 UTC+0000
> >> 0xfffffa80030f09d0 smss.exe 220 4 4 31
> >> ------ 0 2013-07-16 12:24:33 UTC+0000
> >> 0xfffffa80034574d0 csrss.exe 300 292 9 339
> >> 0 0 2013-07-16 12:24:34 UTC+0000
> >> 0xfffffa8003465b30 wininit.exe 352 292 7 93
> >> 0 0 2013-07-16 12:24:34 UTC+0000
> >> 0xfffffa8003469b30 csrss.exe 368 344 8 76
> >> 1 0 2013-07-16 12:24:34 UTC+0000
> >> 0xfffffa800349c280 winlogon.exe 412 344 5 83
> >> 1 0 2013-07-16 12:24:34 UTC+0000
> >> 0xfffffa80034a7160 services.exe 448 352 17 215
> >> 0 0 2013-07-16 12:24:34 UTC+0000
> >> 0xfffffa80034b4b30 lsass.exe 464 352 9 458
> >> 0 0 2013-07-16 12:24:34 UTC+0000
> >> 0xfffffa80034b64f0 lsm.exe 472 352 12 194
> >> 0 0 2013-07-16 12:24:34 UTC+0000
> >> 0xfffffa800350cb30 svchost.exe 584 448 17 355
> >> 0 0 2013-07-16 12:24:34 UTC+0000
> >> 0xfffffa8003522060 svchost.exe 664 448 13 221
> >> 0 0 2013-07-16 12:24:34 UTC+0000
> >> 0xfffffa8003547060 svchost.exe 724 448 16 312
> >> 0 0 2013-07-16 12:24:34 UTC+0000
> >> 0xfffffa8003552b30 LogonUI.exe 744 412 8 157
> >> 1 0 2013-07-16 12:24:34 UTC+0000
> >> 0xfffffa8003572b30 svchost.exe 812 448 43 782
> >> 0 0 2013-07-16 12:24:34 UTC+0000
> >> 0xfffffa8003594b30 svchost.exe 856 448 14 234
> >> 0 0 2013-07-16 12:24:34 UTC+0000
> >> 0xfffffa800359b9b0 svchost.exe 900 448 8 128
> >> 0 0 2013-07-16 12:24:34 UTC+0000
> >> 0xfffffa80035b3060 svchost.exe 940 448 19 361
> >> 0 0 2013-07-16 12:24:34 UTC+0000
> >> 0xfffffa80035fcb30 svchost.exe 372 448 16 259
> >> 0 0 2013-07-16 12:24:35 UTC+0000
> >> 0xfffffa80035f6b30 spoolsv.exe 1048 448 8 89
> >> 0 0 2013-07-16 12:24:35 UTC+0000
> >> 0xfffffa8003679650 blnsvr.exe 1076 448 7 100
> >> 0 0 2013-07-16 12:24:35 UTC+0000
> >> 0xfffffa80035e5450 svchost.exe 1116 448 4 50
> >> 0 0 2013-07-16 12:24:35 UTC+0000
> >> 0xfffffa8003732b30 WmiPrvSE.exe 1364 584 15 294
> >> 0 0 2013-07-16 12:24:35 UTC+0000
> >> 0xfffffa8003767250 svchost.exe 1484 448 12 241
> >> 0 0 2013-07-16 12:24:35 UTC+0000
> >> 0xfffffa80037df620 WmiApSrv.exe 1684 448 7 112
> >> 0 0 2013-07-16 12:24:36 UTC+0000
> >> 0xfffffa80037a56c0 WmiPrvSE.exe 1716 584 7 105
> >> 0 0 2013-07-16 12:24:36 UTC+0000
> >> 0xfffffa8003763270 WmiPrvSE.exe 1764 584 7 175
> >> 0 0 2013-07-16 12:24:38 UTC+0000
> >>
> >>
> >> VM memory size: 3588 MB
> >>
> >> Determining profile based on KDBG search...
> >>
> >> Suggested Profile(s) : Win2008R2SP0x64, Win7SP1x64,
> >> Win7SP0x64, Win2008R2SP1x64
> >> AS Layer1 : AMD64PagedMemory (Kernel AS)
> >> AS Layer2 : FileAddressSpace
> >> (/var/lib/libvirt/qemu/save/win-3588.ram)
> >> PAE type : PAE
> >> DTB : 0x187000L
> >> KDBG : 0xf8000180e0a0
> >> Number of Processors : 2
> >> Image Type (Service Pack) : 1
> >> KPCR for CPU 0 : 0xfffff8000180fd00L
> >> KPCR for CPU 1 : 0xfffff880009b8000L
> >> KUSER_SHARED_DATA : 0xfffff78000000000L
> >> Image date and time : 2013-07-16 12:50:59 UTC+0000
> >> Image local date and time : 2013-07-16 12:50:59 +0000
> >>
> >> Offset(V) Name PID PPID Thds Hnds
> >> Sess Wow64 Start Exit
> >> ------------------ -------------------- ------ ------ ------ --------
> >> ------ ------ ------------------------------
> >> ------------------------------
> >> 0xfffffa800308d9e0 System 4 0 68 275
> >> ------ 0 2013-07-16 12:50:55 UTC+0000
> >>
> >>
> >> VM memory size: 3840 MB
> >>
> >> Determining profile based on KDBG search...
> >>
> >> Suggested Profile(s) : Win2008R2SP0x64, Win7SP1x64,
> >> Win7SP0x64, Win2008R2SP1x64
> >> AS Layer1 : AMD64PagedMemory (Kernel AS)
> >> AS Layer2 : FileAddressSpace
> >> (/var/lib/libvirt/qemu/save/win-3840.ram)
> >> PAE type : PAE
> >> DTB : 0x187000L
> >> KDBG : 0xf800018400a0
> >> Number of Processors : 1
> >> Image Type (Service Pack) : 1
> >> KPCR for CPU 0 : 0xfffff80001841d00L
> >> KUSER_SHARED_DATA : 0xfffff78000000000L
> >> Image date and time : 2013-07-16 12:28:55 UTC+0000
> >> Image local date and time : 2013-07-16 12:28:55 +0000
> >>
> >> Offset(V) Name PID PPID Thds Hnds
> >> Sess Wow64 Start Exit
> >> ------------------ -------------------- ------ ------ ------ --------
> >> ------ ------ ------------------------------
> >> ------------------------------
> >> 0xfffffa80033849e0 System 4 0 72 --------
> >> ------ 0 2013-07-16 12:28:47 UTC+0000
> >> _______________________________________________
> >> Vol-users mailing list
> >> Vol-users(a)volatilityfoundation.org
> >> http://lists.volatilityfoundation.org/mailman/listinfo/vol-users
> >
> >
>
12 years
- 1
- 0