Hey Brian,

In a nutshell, we've investigated the issue and found that it's most definitely not LiME causing the problem. LiME is producing a valid memory dump. The problem is with volatility's DTB finding and address-space verification checks (two different factors there). Fortunately, we believe it to be a quite isolated and extraordinary case specific to CentOS 5.3 x64 with the 2.6.18.x kernel. I will explain a little about how we came to this conclusion, however it will mostly be for documentation purposes. After understanding the differences and considering the time involved in supporting this "special" case, we probably won't be supporting it in the future (also since its going on 4+ years old now).

The first task was actually finding install media for CentOS 5.3. I tracked one down here: http://www2.frugalware.org/mirror/centos.org/5.3/isos/x86_64/

After installing it, I built the volatility profile and dumped memory with LiME. When trying to analyze with volatility, I got the same error as you did, the main indicator was:

DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.amd64.AMD64PagedMemory'>
DEBUG1 : volatility.utils : Failed instantiating AMD64PagedMemory: Failed valid Address Space check

The cause of this error message is the VolatilityLinuxIntelValidAS check is failing. You can see the code here: http://code.google.com/p/volatility/source/browse/trunk/volatility/plugins/overlays/linux/linux.py#649. To pass the check, the virtual address of the init_task structure, when translated to a physical address, must equal the virtual address minus a hard-coded "shift" value of 0xffffffff80000000 (which is consistent on all x64 distributions and kernel versions that we've seen.

Before we can translate virtual addresses to physical, we need the kernel DTB. This is found on typical systems by querying the init_level4_pgt symbol from System.map and subtracting the shift value. As you can see, this symbol exists in the CentOS 5.3 x64 2.6.18.x:

# grep init_level4_pgt System.map-2.6.18-128.el5

ffffffff80001000 T init_level4_pgt

When you subtract the shift, it leaves 0x1000. If you view this offset in the *raw* memory dump (not the lime formatted memory dump, because that has headers which will change the offset), you'll see that 0x1000 is most definitely not a DTB, its all zeros:

$ xxd -s 0x1000 ~/Documents/Virtual\ Machines.localized/CentOS53x64.vmwarevm/CentOS53x64-Snapshot1.vmem | less

0001000: 0000 0000 0000 0000 0000 0000 0000 0000 ................

0001010: 0000 0000 0000 0000 0000 0000 0000 0000 ................

0001020: 0000 0000 0000 0000 0000 0000 0000 0000 ................

0001030: 0000 0000 0000 0000 0000 0000 0000 0000 ................

0001040: 0000 0000 0000 0000 0000 0000 0000 0000 ................

0001050: 0000 0000 0000 0000 0000 0000 0000 0000 ................

0001060: 0000 0000 0000 0000 0000 0000 0000 0000 ................

0001070: 0000 0000 0000 0000 0000 0000 0000 0000 ................

After digging through the kernel source, Andrew found that this particular kernel uses a symbol called boot_level4_pgt to store the kernel DTB instead. But:

# grep boot_level4_pgt System.map-2.6.18-128.el5

So, what we have is a System.map that initializes init_level4_pgt to a value that isn't the DTB, and the value that stores the real DTB is not in the map file. We've never heard of boot_level4_pgt, so apparently it was a very short-lived thing.

Anyway, back on the live CentOS 5.3 system, we loaded an lkm that read and printed the value of the init_task structure's mm->pgd value. It came up with 0x3e58d000 for my VM. I went back and plugged that into the --dtb parameter to volatility (which will override the value it gets from init_level4_pgt):

$ python2.6 vol.py -f ~/Desktop/centos53x64-128.lime --profile="LinuxCentOS53-128x64" --dtb=0x3e58d000

There was still an issue with verifying the AMD64PagedMemory address space, however this time the virtual address of the init_task was actually translated properly (since we supplied the DTB manually). Some debug messaged showed that the check was failing because the shift value was 0x200000 bytes off. So a very simple patch of the following nature was able to fix everything up:

Index: volatility/plugins/overlays/linux/linux64.py

===================================================================

--- volatility/plugins/overlays/linux/linux64.py (revision 3020)

+++ volatility/plugins/overlays/linux/linux64.py (working copy)

@@ -33,7 +33,7 @@

"""Tries to locate the DTB."""

profile = self.obj_vm.profile

- yield profile.get_symbol("init_level4_pgt") - 0xffffffff80000000

+ yield profile.get_symbol("init_level4_pgt") - 0xffffffff7fe00000

class Linux64ObjectClasses(obj.ProfileModification):

Index: volatility/plugins/overlays/linux/linux.py

===================================================================

--- volatility/plugins/overlays/linux/linux.py (revision 3020)

+++ volatility/plugins/overlays/linux/linux.py (working copy)

@@ -656,7 +656,7 @@

if self.obj_vm.profile.metadata.get('memory_model', '32bit') == "32bit":

shift = 0xc0000000

else:

- shift = 0xffffffff80000000

+ shift = 0xffffffff7fe00000

yield self.obj_vm.vtop(init_task_addr) == init_task_addr - shift

Now we can find the DTB and pass the AMD64 address space check with no issues:

$ python2.6 vol.py -f ~/Desktop/centos53x64-128.lime --profile="LinuxCentOS53-128x64" linux_pslist

Volatile Systems Volatility Framework 2.3_alpha

Offset Name Pid Uid Gid DTB Start Time

------------------ -------------------- --------------- --------------- ------ ------------------ ----------

......

0x000081003cbbf080 firefox 6905 500 500 0x000000002256a000 Sun, 03 Feb 2013 12:19:45 +0000

0x000081001bf08860 su 7008 500 500 0x0000000018130000 Sun, 03 Feb 2013 12:21:04 +0000

0x000081002d899100 bash 7011 0 0 0x0000000025b19000 Sun, 03 Feb 2013 12:21:05 +0000

0x000081002a3bf820 notification-da 9374 500 500 0x000000001f444000 Sun, 03 Feb 2013 12:26:40 +0000

0x000081003fe6e7a0 bash 10392 500 500 0x0000000014785000 Sun, 03 Feb 2013 12:34:50 +0000

0x0000810014f9a100 insmod 10597 0 0 0x0000000012399000 Sun, 03 Feb 2013 12:36:11 +0000

$ python2.6 vol.py -f ~/Desktop/centos53x64-128.lime --profile="LinuxCentOS53-128x64" linux_dmesg

Volatile Systems Volatility Framework 2.3_alpha

WARNING : volatility.obj : Overlay structure cpuinfo_x86 not present in vtypes

<5>Linux version 2.6.18-128.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) #1 SMP Wed Jan 21 10:41:14 EST 2009

<6>Command line: ro root=LABEL=/ quiet

<6>BIOS-provided physical RAM map:

<4> BIOS-e820: 0000000000000000 - 000000000009f000 (usable)

<4> BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)

<4> BIOS-e820: 00000000000ca000 - 00000000000cc000 (reserved)

$ python2.6 vol.py -f ~/Desktop/centos53x64-128.lime --profile="LinuxCentOS53-128x64" linux_ifconfig

Volatile Systems Volatility Framework 2.3_alpha

WARNING : volatility.obj : Overlay structure cpuinfo_x86 not present in vtypes

Interface IP Address MAC Address Promiscous Mode

---------------- -------------------- ------------------ ---------------

lo 127.0.0.1 00:00:00:00:00:00 False

eth0 192.168.16.240 00:0c:29:d4:30:9e False

sit0 0.0.0.0 00:00:00:00:00:00 False

One of the reasons the shift *could* be different is if CONFIG_RELOCATABLE kernel option is set. However, on my CentOS 5.3 x64 2.6.18.x, the option is *not* enabled and I'm guessing its not on your either. Plus, Andrew has analyzed systems in the past with this option *enabled* yet the shift value was still 0xffffffff80000000.

So in the future if this becomes a problem, we may consider some fuzzing/brute-forcing around the 0xffffffff80000000 plus or minus 0x200000 area, but currently its only been seen as different on on the 4+ year old CentOS 5.3 x64 2.6.18.x, so its not a major issue for us at this time.

Hope this helps!

MHL

On Thu, Jan 31, 2013 at 11:24 AM, Brian Keefer <chort@effu.se> wrote:

Hello, sending to list for archive(?) sake.

chort@hydra:~/code/profiles-volatility/CentOS_5.3_x64$ vol.py --profile=LinuxCentOS_5_3x64 -f /fun/ir/geriatrix.lime -dd linux_lsmod

Volatile Systems Volatility Framework 2.3_alpha

DEBUG : volatility.utils : Voting round
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.macho.MachOAddressSpace'>
DEBUG1 : volatility.utils : Failed instantiating MachOAddressSpace: mac: need base
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.lime.LimeAddressSpace'>
DEBUG1 : volatility.utils : Failed instantiating LimeAddressSpace: lime: need base
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.hibernate.WindowsHiberFileSpace32'>
DEBUG1 : volatility.utils : Failed instantiating WindowsHiberFileSpace32: No base Address Space
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace64'>
DEBUG1 : volatility.utils : Failed instantiating WindowsCrashDumpSpace64: No base Address Space
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.hpak.HPAKAddressSpace'>
DEBUG1 : volatility.utils : Failed instantiating HPAKAddressSpace: No base Address Space
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.vboxelf.VirtualBoxCoreDumpElf64'>
DEBUG1 : volatility.utils : Failed instantiating VirtualBoxCoreDumpElf64: No base Address Space
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.vmware.VMWareSnapshotFile'>
DEBUG1 : volatility.utils : Failed instantiating VMWareSnapshotFile: No base Address Space
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace32'>
DEBUG1 : volatility.utils : Failed instantiating WindowsCrashDumpSpace32: No base Address Space
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.intel.JKIA32PagedMemoryPae'>
DEBUG1 : volatility.utils : Failed instantiating JKIA32PagedMemoryPae: No base Address Space
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.amd64.AMD64PagedMemory'>
DEBUG1 : volatility.utils : Failed instantiating AMD64PagedMemory: No base Address Space
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.intel.JKIA32PagedMemory'>
DEBUG1 : volatility.utils : Failed instantiating JKIA32PagedMemory: No base Address Space
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.legacyintel.IA32PagedMemoryPae'>
DEBUG1 : volatility.utils : Failed instantiating IA32PagedMemoryPae: Module disabled
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.legacyintel.IA32PagedMemory'>
DEBUG1 : volatility.utils : Failed instantiating IA32PagedMemory: Module disabled
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.standard.FileAddressSpace'>
DEBUG : volatility.plugins.overlays.linux.linux: CentOS_5.3: Found dwarf file System.map-2.6.18-128.el5 with 365 symbols
DEBUG : volatility.plugins.overlays.linux.linux: CentOS_5.3: Found system file System.map-2.6.18-128.el5 with 1 symbols
DEBUG : volatility.obj : Applying modification from BashTypes
DEBUG : volatility.obj : Applying modification from BasicObjectClasses
DEBUG : volatility.obj : Applying modification from ELF64Modification
DEBUG : volatility.obj : Applying modification from HPAKVTypes
DEBUG : volatility.obj : Applying modification from LimeTypes
DEBUG : volatility.obj : Applying modification from MachoTypes
DEBUG : volatility.obj : Applying modification from MbrObjectTypes
DEBUG : volatility.obj : Applying modification from VMwareVTypesModification
DEBUG : volatility.obj : Applying modification from VirtualBoxModification
DEBUG : volatility.obj : Applying modification from LinuxKmemCacheOverlay
DEBUG : volatility.obj : Applying modification from LinuxMountOverlay
DEBUG : volatility.obj : Applying modification from LinuxObjectClasses
DEBUG : volatility.obj : Applying modification from LinuxOverlay

WARNING : volatility.obj : Overlay structure cpuinfo_x86 not present in vtypes

DEBUG : volatility.obj : Applying modification from Linux64ObjectClasses
DEBUG : volatility.utils : Succeeded instantiating <volatility.plugins.addrspaces.standard.FileAddressSpace object at 0x5c93d50>
DEBUG : volatility.utils : Voting round
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.macho.MachOAddressSpace'>
DEBUG1 : volatility.utils : Failed instantiating MachOAddressSpace: MachO Header signature invalid
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.lime.LimeAddressSpace'>
DEBUG1 : volatility.obj : None object instantiated: Invalid Address 0x3FF8F860, instantiating lime_header
DEBUG : volatility.utils : Succeeded instantiating <volatility.plugins.addrspaces.lime.LimeAddressSpace object at 0x5c93d10>
DEBUG : volatility.utils : Voting round
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.macho.MachOAddressSpace'>
DEBUG1 : volatility.utils : Failed instantiating MachOAddressSpace: MachO Header signature invalid
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.lime.LimeAddressSpace'>
DEBUG1 : volatility.utils : Failed instantiating LimeAddressSpace: Invalid Lime header signature
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.hibernate.WindowsHiberFileSpace32'>
DEBUG1 : volatility.utils : Failed instantiating WindowsHiberFileSpace32: PO_MEMORY_IMAGE is not available in profile
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace64'>
DEBUG1 : volatility.utils : Failed instantiating WindowsCrashDumpSpace64: Header signature invalid
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.hpak.HPAKAddressSpace'>
DEBUG1 : volatility.utils : Failed instantiating HPAKAddressSpace: Invalid magic found
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.vboxelf.VirtualBoxCoreDumpElf64'>
DEBUG1 : volatility.utils : Failed instantiating VirtualBoxCoreDumpElf64: ELF64 Header signature invalid
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.vmware.VMWareSnapshotFile'>
DEBUG1 : volatility.utils : Failed instantiating VMWareSnapshotFile: Invalid VMware signature: 0xf000ff53
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace32'>
DEBUG1 : volatility.utils : Failed instantiating WindowsCrashDumpSpace32: Header signature invalid
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.intel.JKIA32PagedMemoryPae'>
DEBUG1 : volatility.utils : Failed instantiating JKIA32PagedMemoryPae: Incompatible profile LinuxCentOS_5_3x64 selected
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.amd64.AMD64PagedMemory'>
DEBUG1 : volatility.utils : Failed instantiating AMD64PagedMemory: Failed valid Address Space check
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.intel.JKIA32PagedMemory'>
DEBUG1 : volatility.utils : Failed instantiating JKIA32PagedMemory: Incompatible profile LinuxCentOS_5_3x64 selected
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.legacyintel.IA32PagedMemoryPae'>
DEBUG1 : volatility.utils : Failed instantiating IA32PagedMemoryPae: Module disabled
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.legacyintel.IA32PagedMemory'>
DEBUG1 : volatility.utils : Failed instantiating IA32PagedMemory: Module disabled
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.standard.FileAddressSpace'>
DEBUG1 : volatility.utils : Failed instantiating FileAddressSpace: Must be first Address Space
DEBUG : volatility.utils : Trying <class 'volatility.plugins.addrspaces.arm.ArmAddressSpace'>
DEBUG1 : volatility.utils : Failed instantiating ArmAddressSpace: Incompatible profile LinuxCentOS_5_3x64 selected

No suitable address space mapping found
Tried to open image as:
MachOAddressSpace: mac: need base
LimeAddressSpace: lime: need base
WindowsHiberFileSpace32: No base Address Space
WindowsCrashDumpSpace64: No base Address Space
HPAKAddressSpace: No base Address Space
VirtualBoxCoreDumpElf64: No base Address Space
VMWareSnapshotFile: No base Address Space
WindowsCrashDumpSpace32: No base Address Space
JKIA32PagedMemoryPae: No base Address Space
AMD64PagedMemory: No base Address Space
JKIA32PagedMemory: No base Address Space
IA32PagedMemoryPae: Module disabled
IA32PagedMemory: Module disabled
MachOAddressSpace: MachO Header signature invalid
MachOAddressSpace: MachO Header signature invalid
LimeAddressSpace: Invalid Lime header signature
WindowsHiberFileSpace32: PO_MEMORY_IMAGE is not available in profile
WindowsCrashDumpSpace64: Header signature invalid
HPAKAddressSpace: Invalid magic found
VirtualBoxCoreDumpElf64: ELF64 Header signature invalid
VMWareSnapshotFile: Invalid VMware signature: 0xf000ff53
WindowsCrashDumpSpace32: Header signature invalid
JKIA32PagedMemoryPae: Incompatible profile LinuxCentOS_5_3x64 selected
AMD64PagedMemory: Failed valid Address Space check
JKIA32PagedMemory: Incompatible profile LinuxCentOS_5_3x64 selected
IA32PagedMemoryPae: Module disabled
IA32PagedMemory: Module disabled
FileAddressSpace: Must be first Address Space
ArmAddressSpace: Incompatible profile LinuxCentOS_5_3x64 selected

--
chort

On Jan 31, 2013, at 8:20 AM, Andrew Case wrote:

> Hey,
>
> Can you run again with "-dd" added before linux_lsmod and send me the output?
>
> The lack of cpuinfo_x86 does not change lsmod so do not worry about that part...
>
> On Thu, Jan 31, 2013 at 1:18 AM, Brian Keefer <chort@effu.se> wrote:
>> I built LiME from the tarball on the project site (not latest svn) and was able to dump memory successfully (type=lime). After many trials and tribulations I was able to get the Volatility profile built for CentOS 5.3x64 (had to remove pmem from the Makefile). I put the profile in the correct directory, and vol.py --info lists it as expected, however when I try to use the profile with the memory image I get an error.
>>
>> chort@hydra:~/code/profiles-volatility/CentOS_5.3_x64$ vol.py --profile=LinuxCentOS_5_3x64 -f /fun/ir/geriatrix.lime linux_lsmod
>> Volatile Systems Volatility Framework 2.3_alpha
>> WARNING : volatility.obj : Overlay structure cpuinfo_x86 not present in vtypes
>> No suitable address space mapping found
>> Tried to open image as:
>> MachOAddressSpace: mac: need base
>> LimeAddressSpace: lime: need base
>> WindowsHiberFileSpace32: No base Address Space
>> WindowsCrashDumpSpace64: No base Address Space
>> HPAKAddressSpace: No base Address Space
>> VirtualBoxCoreDumpElf64: No base Address Space
>> VMWareSnapshotFile: No base Address Space
>> WindowsCrashDumpSpace32: No base Address Space
>> JKIA32PagedMemoryPae: No base Address Space
>> AMD64PagedMemory: No base Address Space
>> JKIA32PagedMemory: No base Address Space
>> IA32PagedMemoryPae: Module disabled
>> IA32PagedMemory: Module disabled
>> MachOAddressSpace: MachO Header signature invalid
>> MachOAddressSpace: MachO Header signature invalid
>> LimeAddressSpace: Invalid Lime header signature
>> WindowsHiberFileSpace32: PO_MEMORY_IMAGE is not available in profile
>> WindowsCrashDumpSpace64: Header signature invalid
>> HPAKAddressSpace: Invalid magic found
>> VirtualBoxCoreDumpElf64: ELF64 Header signature invalid
>> VMWareSnapshotFile: Invalid VMware signature: 0xf000ff53
>> WindowsCrashDumpSpace32: Header signature invalid
>> JKIA32PagedMemoryPae: Incompatible profile LinuxCentOS_5_3x64 selected
>> AMD64PagedMemory: Failed valid Address Space check
>> JKIA32PagedMemory: Incompatible profile LinuxCentOS_5_3x64 selected
>> IA32PagedMemoryPae: Module disabled
>> IA32PagedMemory: Module disabled
>> FileAddressSpace: Must be first Address Space
>> ArmAddressSpace: Incompatible profile LinuxCentOS_5_3x64 selected
>>
>> On a hunch I checked the directory I built the profile in (copied headers & source from the target system):
>> chort@hydra:~/code/profiles-volatility/CentOS_5.3_x64$ grep cpuinfo *
>> System.map-2.6.18-128.el5:ffffffff8006f328 t show_cpuinfo
>> System.map-2.6.18-128.el5:ffffffff80103251 t cpuinfo_open
>> System.map-2.6.18-128.el5:ffffffff8020eadb t show_cpuinfo_max_freq
>> System.map-2.6.18-128.el5:ffffffff8020eafa t show_cpuinfo_min_freq
>> System.map-2.6.18-128.el5:ffffffff8020f759 t show_cpuinfo_cur_freq
>> System.map-2.6.18-128.el5:ffffffff802f0bc0 D cpuinfo_op
>> System.map-2.6.18-128.el5:ffffffff80308420 d proc_cpuinfo_operations
>> System.map-2.6.18-128.el5:ffffffff803319a0 d cpuinfo_cur_freq
>> System.map-2.6.18-128.el5:ffffffff80331b20 d cpuinfo_min_freq
>> System.map-2.6.18-128.el5:ffffffff80331b60 d cpuinfo_max_freq
>>
>>
>> Platform running Volatility (2.3_alpha, latest from svn):
>> Linux hydra 3.2.0-35-generic #55-Ubuntu SMP Wed Dec 5 17:42:16 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
>>
>> Source of memory image:
>> Linux geriatrix.smtps.net 2.6.18-128.el5 #1 SMP Wed Jan 21 10:41:14 EST 2009 x86_64 x86_64 x86_64 GNU/Linux
>>
>> What am I missing?
>>
>>
>> --
>> chort
>>
>>
>>
>> _______________________________________________
>> Vol-users mailing list
>> Vol-users@volatilesystems.com
>> http://lists.volatilityfoundation.org/mailman/listinfo/vol-users

_______________________________________________
Vol-users mailing list
Vol-users@volatilesystems.com
http://lists.volatilityfoundation.org/mailman/listinfo/vol-users