Re: [Vol-users] Linux profile w/LiME not working

Hey Brian, In a nutshell, we've investigated the issue and found that it's most definitely not LiME causing the problem. LiME is producing a valid memory dump. The problem is with volatility's DTB finding and address-space verification checks (two different factors there). Fortunately, we believe it to be a quite isolated and extraordinary case specific to CentOS 5.3 x64 with the 2.6.18.x kernel. I will explain a little about how we came to this conclusion, however it will mostly be for documentation purposes. After understanding the differences and considering the time involved in supporting this "special" case, we probably won't be supporting it in the future (also since its going on 4+ years old now). The first task was actually finding install media for CentOS 5.3. I tracked one down here: http://www2.frugalware.org/mirror/centos.org/5.3/isos/x86_64/ After installing it, I built the volatility profile and dumped memory with LiME. When trying to analyze with volatility, I got the same error as you did, the main indicator was: DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.amd64.AMD64PagedMemory'> DEBUG1 : volatility.utils : Failed instantiating AMD64PagedMemory: Failed valid Address Space check The cause of this error message is the VolatilityLinuxIntelValidAS check is failing. You can see the code here: http://code.google.com/p/volatility/source/browse/trunk/volatility/plugins/…. To pass the check, the virtual address of the init_task structure, when translated to a physical address, must equal the virtual address minus a hard-coded "shift" value of 0xffffffff80000000 (which is consistent on all x64 distributions and kernel versions that we've seen. Before we can translate virtual addresses to physical, we need the kernel DTB. This is found on typical systems by querying the init_level4_pgt symbol from System.map and subtracting the shift value. As you can see, this symbol exists in the CentOS 5.3 x64 2.6.18.x: # grep init_level4_pgt System.map-2.6.18-128.el5 ffffffff80001000 T init_level4_pgt When you subtract the shift, it leaves 0x1000. If you view this offset in the *raw* memory dump (not the lime formatted memory dump, because that has headers which will change the offset), you'll see that 0x1000 is most definitely not a DTB, its all zeros: $ xxd -s 0x1000 ~/Documents/Virtual\ Machines.localized/CentOS53x64.vmwarevm/CentOS53x64-Snapshot1.vmem | less 0001000: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0001010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0001020: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0001030: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0001040: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0001050: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0001060: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0001070: 0000 0000 0000 0000 0000 0000 0000 0000 ................ After digging through the kernel source, Andrew found that this particular kernel uses a symbol called boot_level4_pgt to store the kernel DTB instead. But: # grep boot_level4_pgt System.map-2.6.18-128.el5 # So, what we have is a System.map that initializes init_level4_pgt to a value that isn't the DTB, and the value that stores the real DTB is not in the map file. We've never heard of boot_level4_pgt, so apparently it was a very short-lived thing. Anyway, back on the live CentOS 5.3 system, we loaded an lkm that read and printed the value of the init_task structure's mm->pgd value. It came up with 0x3e58d000 for my VM. I went back and plugged that into the --dtb parameter to volatility (which will override the value it gets from init_level4_pgt): $ python2.6 vol.py -f ~/Desktop/centos53x64-128.lime --profile="LinuxCentOS53-128x64" --dtb=0x3e58d000 There was still an issue with verifying the AMD64PagedMemory address space, however this time the virtual address of the init_task was actually translated properly (since we supplied the DTB manually). Some debug messaged showed that the check was failing because the shift value was 0x200000 bytes off. So a very simple patch of the following nature was able to fix everything up: Index: volatility/plugins/overlays/linux/linux64.py =================================================================== --- volatility/plugins/overlays/linux/linux64.py (revision 3020) +++ volatility/plugins/overlays/linux/linux64.py (working copy) @@ -33,7 +33,7 @@ """Tries to locate the DTB.""" profile = self.obj_vm.profile - yield profile.get_symbol("init_level4_pgt") - 0xffffffff80000000 + yield profile.get_symbol("init_level4_pgt") - 0xffffffff7fe00000 class Linux64ObjectClasses(obj.ProfileModification): Index: volatility/plugins/overlays/linux/linux.py =================================================================== --- volatility/plugins/overlays/linux/linux.py (revision 3020) +++ volatility/plugins/overlays/linux/linux.py (working copy) @@ -656,7 +656,7 @@ if self.obj_vm.profile.metadata.get('memory_model', '32bit') == "32bit": shift = 0xc0000000 else: - shift = 0xffffffff80000000 + shift = 0xffffffff7fe00000 yield self.obj_vm.vtop(init_task_addr) == init_task_addr - shift Now we can find the DTB and pass the AMD64 address space check with no issues: $ python2.6 vol.py -f ~/Desktop/centos53x64-128.lime --profile="LinuxCentOS53-128x64" linux_pslist Volatile Systems Volatility Framework 2.3_alpha Offset Name Pid Uid Gid DTB Start Time ------------------ -------------------- --------------- --------------- ------ ------------------ ---------- ...... 0x000081003cbbf080 firefox 6905 500 500 0x000000002256a000 Sun, 03 Feb 2013 12:19:45 +0000 0x000081001bf08860 su 7008 500 500 0x0000000018130000 Sun, 03 Feb 2013 12:21:04 +0000 0x000081002d899100 bash 7011 0 0 0x0000000025b19000 Sun, 03 Feb 2013 12:21:05 +0000 0x000081002a3bf820 notification-da 9374 500 500 0x000000001f444000 Sun, 03 Feb 2013 12:26:40 +0000 0x000081003fe6e7a0 bash 10392 500 500 0x0000000014785000 Sun, 03 Feb 2013 12:34:50 +0000 0x0000810014f9a100 insmod 10597 0 0 0x0000000012399000 Sun, 03 Feb 2013 12:36:11 +0000 $ python2.6 vol.py -f ~/Desktop/centos53x64-128.lime --profile="LinuxCentOS53-128x64" linux_dmesg Volatile Systems Volatility Framework 2.3_alpha WARNING : volatility.obj : Overlay structure cpuinfo_x86 not present in vtypes <5>Linux version 2.6.18-128.el5 (mockbuild(a)builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) #1 SMP Wed Jan 21 10:41:14 EST 2009 <6>Command line: ro root=LABEL=/ quiet <6>BIOS-provided physical RAM map: <4> BIOS-e820: 0000000000000000 - 000000000009f000 (usable) <4> BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved) <4> BIOS-e820: 00000000000ca000 - 00000000000cc000 (reserved) $ python2.6 vol.py -f ~/Desktop/centos53x64-128.lime --profile="LinuxCentOS53-128x64" linux_ifconfig Volatile Systems Volatility Framework 2.3_alpha WARNING : volatility.obj : Overlay structure cpuinfo_x86 not present in vtypes Interface IP Address MAC Address Promiscous Mode ---------------- -------------------- ------------------ --------------- lo 127.0.0.1 00:00:00:00:00:00 False eth0 192.168.16.240 00:0c:29:d4:30:9e False sit0 0.0.0.0 00:00:00:00:00:00 False One of the reasons the shift *could* be different is if CONFIG_RELOCATABLE kernel option is set. However, on my CentOS 5.3 x64 2.6.18.x, the option is *not* enabled and I'm guessing its not on your either. Plus, Andrew has analyzed systems in the past with this option *enabled* yet the shift value was still 0xffffffff80000000. So in the future if this becomes a problem, we may consider some fuzzing/brute-forcing around the 0xffffffff80000000 plus or minus 0x200000 area, but currently its only been seen as different on on the 4+ year old CentOS 5.3 x64 2.6.18.x, so its not a major issue for us at this time. Hope this helps! MHL On Thu, Jan 31, 2013 at 11:24 AM, Brian Keefer &lt;chort(a)effu.se&gt; wrote: Hello, sending to list for archive(?) sake. chort@hydra:~/code/profiles-volatility/CentOS_5.3_x64$ vol.py --profile=LinuxCentOS_5_3x64 -f /fun/ir/geriatrix.lime -dd linux_lsmod Volatile Systems Volatility Framework 2.3_alpha DEBUG : volatility.utils : Voting round DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.macho.MachOAddressSpace'> DEBUG1 : volatility.utils : Failed instantiating MachOAddressSpace: mac: need base DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.lime.LimeAddressSpace'> DEBUG1 : volatility.utils : Failed instantiating LimeAddressSpace: lime: need base DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.hibernate.WindowsHiberFileSpace32'> DEBUG1 : volatility.utils : Failed instantiating WindowsHiberFileSpace32: No base Address Space DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace64'> DEBUG1 : volatility.utils : Failed instantiating WindowsCrashDumpSpace64: No base Address Space DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.hpak.HPAKAddressSpace'> DEBUG1 : volatility.utils : Failed instantiating HPAKAddressSpace: No base Address Space DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.vboxelf.VirtualBoxCoreDumpElf64'> DEBUG1 : volatility.utils : Failed instantiating VirtualBoxCoreDumpElf64: No base Address Space DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.vmware.VMWareSnapshotFile'> DEBUG1 : volatility.utils : Failed instantiating VMWareSnapshotFile: No base Address Space DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace32'> DEBUG1 : volatility.utils : Failed instantiating WindowsCrashDumpSpace32: No base Address Space DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.intel.JKIA32PagedMemoryPae'> DEBUG1 : volatility.utils : Failed instantiating JKIA32PagedMemoryPae: No base Address Space DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.amd64.AMD64PagedMemory'> DEBUG1 : volatility.utils : Failed instantiating AMD64PagedMemory: No base Address Space DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.intel.JKIA32PagedMemory'> DEBUG1 : volatility.utils : Failed instantiating JKIA32PagedMemory: No base Address Space DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.legacyintel.IA32PagedMemoryPae'> DEBUG1 : volatility.utils : Failed instantiating IA32PagedMemoryPae: Module disabled DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.legacyintel.IA32PagedMemory'> DEBUG1 : volatility.utils : Failed instantiating IA32PagedMemory: Module disabled DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.standard.FileAddressSpace'> DEBUG : volatility.plugins.overlays.linux.linux: CentOS_5.3: Found dwarf file System.map-2.6.18-128.el5 with 365 symbols DEBUG : volatility.plugins.overlays.linux.linux: CentOS_5.3: Found system file System.map-2.6.18-128.el5 with 1 symbols DEBUG : volatility.obj : Applying modification from BashTypes DEBUG : volatility.obj : Applying modification from BasicObjectClasses DEBUG : volatility.obj : Applying modification from ELF64Modification DEBUG : volatility.obj : Applying modification from HPAKVTypes DEBUG : volatility.obj : Applying modification from LimeTypes DEBUG : volatility.obj : Applying modification from MachoTypes DEBUG : volatility.obj : Applying modification from MbrObjectTypes DEBUG : volatility.obj : Applying modification from VMwareVTypesModification DEBUG : volatility.obj : Applying modification from VirtualBoxModification DEBUG : volatility.obj : Applying modification from LinuxKmemCacheOverlay DEBUG : volatility.obj : Applying modification from LinuxMountOverlay DEBUG : volatility.obj : Applying modification from LinuxObjectClasses DEBUG : volatility.obj : Applying modification from LinuxOverlay WARNING : volatility.obj : Overlay structure cpuinfo_x86 not present in vtypes DEBUG : volatility.obj : Applying modification from Linux64ObjectClasses DEBUG : volatility.utils : Succeeded instantiating <volatility.plugins.addrspaces.standard.FileAddressSpace object at 0x5c93d50> DEBUG : volatility.utils : Voting round DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.macho.MachOAddressSpace'> DEBUG1 : volatility.utils : Failed instantiating MachOAddressSpace: MachO Header signature invalid DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.lime.LimeAddressSpace'> DEBUG1 : volatility.obj : None object instantiated: Invalid Address 0x3FF8F860, instantiating lime_header DEBUG : volatility.utils : Succeeded instantiating <volatility.plugins.addrspaces.lime.LimeAddressSpace object at 0x5c93d10> DEBUG : volatility.utils : Voting round DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.macho.MachOAddressSpace'> DEBUG1 : volatility.utils : Failed instantiating MachOAddressSpace: MachO Header signature invalid DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.lime.LimeAddressSpace'> DEBUG1 : volatility.utils : Failed instantiating LimeAddressSpace: Invalid Lime header signature DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.hibernate.WindowsHiberFileSpace32'> DEBUG1 : volatility.utils : Failed instantiating WindowsHiberFileSpace32: PO_MEMORY_IMAGE is not available in profile DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace64'> DEBUG1 : volatility.utils : Failed instantiating WindowsCrashDumpSpace64: Header signature invalid DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.hpak.HPAKAddressSpace'> DEBUG1 : volatility.utils : Failed instantiating HPAKAddressSpace: Invalid magic found DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.vboxelf.VirtualBoxCoreDumpElf64'> DEBUG1 : volatility.utils : Failed instantiating VirtualBoxCoreDumpElf64: ELF64 Header signature invalid DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.vmware.VMWareSnapshotFile'> DEBUG1 : volatility.utils : Failed instantiating VMWareSnapshotFile: Invalid VMware signature: 0xf000ff53 DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace32'> DEBUG1 : volatility.utils : Failed instantiating WindowsCrashDumpSpace32: Header signature invalid DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.intel.JKIA32PagedMemoryPae'> DEBUG1 : volatility.utils : Failed instantiating JKIA32PagedMemoryPae: Incompatible profile LinuxCentOS_5_3x64 selected DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.amd64.AMD64PagedMemory'> DEBUG1 : volatility.utils : Failed instantiating AMD64PagedMemory: Failed valid Address Space check DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.intel.JKIA32PagedMemory'> DEBUG1 : volatility.utils : Failed instantiating JKIA32PagedMemory: Incompatible profile LinuxCentOS_5_3x64 selected DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.legacyintel.IA32PagedMemoryPae'> DEBUG1 : volatility.utils : Failed instantiating IA32PagedMemoryPae: Module disabled DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.legacyintel.IA32PagedMemory'> DEBUG1 : volatility.utils : Failed instantiating IA32PagedMemory: Module disabled DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.standard.FileAddressSpace'> DEBUG1 : volatility.utils : Failed instantiating FileAddressSpace: Must be first Address Space DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.arm.ArmAddressSpace'> DEBUG1 : volatility.utils : Failed instantiating ArmAddressSpace: Incompatible profile LinuxCentOS_5_3x64 selected No suitable address space mapping found Tried to open image as: MachOAddressSpace: mac: need base LimeAddressSpace: lime: need base WindowsHiberFileSpace32: No base Address Space WindowsCrashDumpSpace64: No base Address Space HPAKAddressSpace: No base Address Space VirtualBoxCoreDumpElf64: No base Address Space VMWareSnapshotFile: No base Address Space WindowsCrashDumpSpace32: No base Address Space JKIA32PagedMemoryPae: No base Address Space AMD64PagedMemory: No base Address Space JKIA32PagedMemory: No base Address Space IA32PagedMemoryPae: Module disabled IA32PagedMemory: Module disabled MachOAddressSpace: MachO Header signature invalid MachOAddressSpace: MachO Header signature invalid LimeAddressSpace: Invalid Lime header signature WindowsHiberFileSpace32: PO_MEMORY_IMAGE is not available in profile WindowsCrashDumpSpace64: Header signature invalid HPAKAddressSpace: Invalid magic found VirtualBoxCoreDumpElf64: ELF64 Header signature invalid VMWareSnapshotFile: Invalid VMware signature: 0xf000ff53 WindowsCrashDumpSpace32: Header signature invalid JKIA32PagedMemoryPae: Incompatible profile LinuxCentOS_5_3x64 selected AMD64PagedMemory: Failed valid Address Space check JKIA32PagedMemory: Incompatible profile LinuxCentOS_5_3x64 selected IA32PagedMemoryPae: Module disabled IA32PagedMemory: Module disabled FileAddressSpace: Must be first Address Space ArmAddressSpace: Incompatible profile LinuxCentOS_5_3x64 selected -- chort On Jan 31, 2013, at 8:20 AM, Andrew Case wrote: _______________________________________________ Vol-users mailing list Vol-users(a)volatilityfoundation.org http://lists.volatilityfoundation.org/mailman/listinfo/vol-users