Hey Pasquale, 

Sorry for the delay, I suspect once everyone returns from BsidesSF and RSA, we'll be able to help a bit more. In the meantime I put up some of my notes for compiling android kernels on mac here: http://code.google.com/p/volatility/wiki/AndroidMemoryForensics and you definitely need the case-sensitive file system. 

Talk to you soon,
MHL


On Tue, Feb 26, 2013 at 11:18 AM, Pasquale Stirparo <pstirparo@gmail.com> wrote:
Hi,

Michael thank you for your email. 
I've spent the last two days fighting recompiling the kernel (never got so many errors :( ). At the end most of the errors were due to case-insensitive fs and some missin headers from the HTC kernel source. Still getting final error when trying to finalize the image, but at least the tmp_System.map has been created.
So, I tried to use the System.map created by cross-compiling the kernel

$ make ARCH=arm CROSS_COMPILE= ~/android-ndk-r8d/toolchains/arm-linux-androideabi-4.4.3/prebuilt/darwin-x86/bin/arm-linux-androideabi-

But when I try to run volatility using the profile created I get, for whatever plugins I try to use, the following warning but no output whatsoever:
Volatile Systems Volatility Framework 2.3_alpha
WARNING : volatility.obj      : Overlay structure cpuinfo_x86 not present in vtypes


Regarding kallsyms file, as you wrote I have at the end several entries with 4 arguments each, none of them with "[lime]" though. I tried to remove those anyway and this is the output I got when using such modified kallsyms file:
Volatile Systems Volatility Framework 2.3_alpha
WARNING : volatility.obj      : Overlay structure cpuinfo_x86 not present in vtypes
Offset     Name                 Pid             Uid             Start Time
---------- -------------------- --------------- --------------- ----------
No suitable address space mapping found
Tried to open image as:
 MachOAddressSpace: mac: need base
 LimeAddressSpace: lime: need base
 WindowsHiberFileSpace32: No base Address Space
 WindowsCrashDumpSpace64: No base Address Space
 WindowsCrashDumpSpace32: No base Address Space
 JKIA32PagedMemoryPae: No base Address Space
 AMD64PagedMemory: No base Address Space
 JKIA32PagedMemory: No base Address Space
 IA32PagedMemoryPae: Module disabled
 IA32PagedMemory: Module disabled
 MachOAddressSpace: MachO Header signature invalid
 MachOAddressSpace: MachO Header signature invalid
 LimeAddressSpace: Invalid Lime header signature
 WindowsHiberFileSpace32: PO_MEMORY_IMAGE is not available in profile
 WindowsCrashDumpSpace64: Header signature invalid
 WindowsCrashDumpSpace32: Header signature invalid
 JKIA32PagedMemoryPae - EXCEPTION: unsupported operand type(s) for -: 'NoneType' and 'int'
 AMD64PagedMemory: Incompatible profile LinuxHTCOneXx86 selected
 JKIA32PagedMemory - EXCEPTION: unsupported operand type(s) for -: 'NoneType' and 'int'
 IA32PagedMemoryPae: Module disabled
 IA32PagedMemory: Module disabled
 FileAddressSpace: Must be first Address Space
 ArmAddressSpace - EXCEPTION: unsupported operand type(s) for -: 'NoneType' and 'int'


I'm attaching both the System.map and the Kallsyms files. As I wrote in the previous email, I start to think (even more now) that maybe the lime dumps are not correct (see hexdump attached in my previous email).

Any hint?



On Sat, Feb 23, 2013 at 10:45 PM, Michael Hale Ligh <michael.hale@gmail.com> wrote:
Hey Pasquale, 

So the "ValueError: too many values to unpack" error that you've been getting in both volatility branches (although one branch masks the full backtrace and the other includes it) is due to using /proc/kallsyms. If you look at the line right before the error:

 (str_addr, symbol_type, symbol) = line.strip().split()

There are supposed to be three space separated values in the symbol file. Now if you cat your /proc/kallsyms:

# cat /proc/kallsyms
....
c02f6680 r __param_disable
c02f6694 r __param_pool_mode
c02f66a8 R __stop___param
c02f7000 D _etext
00000000 a tcp.c    [lime]
bf000000 t $a    [lime]
bf000160 t $d    [lime]
bf000168 t $a    [lime]
bf0001e8 t $d    [lime]
bf0001ec t $a    [lime]

In the lines attributed to lime, there are actually 4 values, which raises the ValueError. Aside from this problem, and despite the name "kallsyms", the /proc/kallsyms file doesn't actually contain "all" symbols. For example:

$ grep init_task ~/source/kernel/goldfish/System.map
c02e3000 r __ksymtab_init_task
c02e91f8 r __kstrtab_init_task
c02fae08 D init_task
c031afe8 B init_task_group

# grep init_task /proc/kallsyms                             
c02e3000 r __ksymtab_init_task
c02e91f8 r __kstrtab_init_task

So compared to the kernel's System.map, the /proc/kallsyms is actually missing many symbols. I would suggest building your Volatility profile with the kernel's System.map in this case. Only as a last resort use /proc/kallsyms and even in that case, delete the lines related to lime so it doesn't raise ValueError. 

MHL



On Fri, Feb 22, 2013 at 11:12 AM, Pasquale Stirparo <pstirparo@gmail.com> wrote:
Ok, trying to get my head around this. I get the same error with both vol-2.3_alpha and the latest volatility version available from git repository. There are different erros for different plugins, of course. I started considering that, although if I run strings on the dump I find the data I'm looking for, there could be an error in the dump especially after the error from the linux_arp (below). So running hexdump -C on both my test-dump and the sample-Evo4G one, mine it looks a bit weird even though starts with the right LiME signature. I attached in the txt file the first lines of hexdump, while here there are the two error messages.

Here the two from linux_arp and linux_pslist. Any clue?

hydra:volatility-read-only paco$ python vol.py --profile=LinuxHTCOneXx86 -f /Users/paco/Documents/htconex.lime linux_arp
Volatile Systems Volatility Framework 2.3_alpha
No suitable address space mapping found
Tried to open image as:
 MachOAddressSpace: mac: need base
 LimeAddressSpace: lime: need base
 WindowsHiberFileSpace32: No base Address Space
 WindowsCrashDumpSpace64: No base Address Space
 HPAKAddressSpace: No base Address Space
 VirtualBoxCoreDumpElf64: No base Address Space
 VMWareSnapshotFile: No base Address Space
 WindowsCrashDumpSpace32: No base Address Space
 JKIA32PagedMemoryPae: No base Address Space
 AMD64PagedMemory: No base Address Space
 JKIA32PagedMemory: No base Address Space
 IA32PagedMemoryPae: Module disabled
 IA32PagedMemory: Module disabled
 FileAddressSpace - EXCEPTION: too many values to unpack
 ArmAddressSpace: No base Address Space


hydra:Volatility 2.3-devel paco$ python vol.py --profile=LinuxHTCOneXx86 -f /Users/paco/Documents/htconex.lime linux_pslist
Volatile Systems Volatility Framework 2.3_alpha
Traceback (most recent call last):
  File "vol.py", line 186, in <module>
    main()
  File "vol.py", line 177, in main
    command.execute()
  File "/Users/paco/Documents/Projects/MobiLeak/Paper2-memory/memdump/sample-Rodeo2012/Volatility 2.3-devel/volatility/plugins/linux/common.py", line 51, in execute
    commands.Command.execute(self, *args, **kwargs)
  File "/Users/paco/Documents/Projects/MobiLeak/Paper2-memory/memdump/sample-Rodeo2012/Volatility 2.3-devel/volatility/commands.py", line 111, in execute
    func(outfd, data)
  File "/Users/paco/Documents/Projects/MobiLeak/Paper2-memory/memdump/sample-Rodeo2012/Volatility 2.3-devel/volatility/plugins/linux/pslist.py", line 60, in render_text
    ("Start Time", "")])
  File "/Users/paco/Documents/Projects/MobiLeak/Paper2-memory/memdump/sample-Rodeo2012/Volatility 2.3-devel/volatility/commands.py", line 172, in table_header
    profile = addrspace.BufferAddressSpace(self._config).profile
  File "/Users/paco/Documents/Projects/MobiLeak/Paper2-memory/memdump/sample-Rodeo2012/Volatility 2.3-devel/volatility/addrspace.py", line 169, in __init__
    BaseAddressSpace.__init__(self, None, config, **kwargs)
  File "/Users/paco/Documents/Projects/MobiLeak/Paper2-memory/memdump/sample-Rodeo2012/Volatility 2.3-devel/volatility/addrspace.py", line 71, in __init__
    self.profile = self._set_profile(config.PROFILE)
  File "/Users/paco/Documents/Projects/MobiLeak/Paper2-memory/memdump/sample-Rodeo2012/Volatility 2.3-devel/volatility/addrspace.py", line 94, in _set_profile
    ret = profs[profile_name]()
  File "/Users/paco/Documents/Projects/MobiLeak/Paper2-memory/memdump/sample-Rodeo2012/Volatility 2.3-devel/volatility/plugins/overlays/linux/linux.py", line 145, in __init__
    obj.Profile.__init__(self, *args, **kwargs)
  File "/Users/paco/Documents/Projects/MobiLeak/Paper2-memory/memdump/sample-Rodeo2012/Volatility 2.3-devel/volatility/obj.py", line 857, in __init__
    self.reset()
  File "/Users/paco/Documents/Projects/MobiLeak/Paper2-memory/memdump/sample-Rodeo2012/Volatility 2.3-devel/volatility/plugins/overlays/linux/linux.py", line 156, in reset
    self.load_sysmap()
  File "/Users/paco/Documents/Projects/MobiLeak/Paper2-memory/memdump/sample-Rodeo2012/Volatility 2.3-devel/volatility/plugins/overlays/linux/linux.py", line 199, in load_sysmap
    _memmodel, sysmapvar = parse_system_map(sysmapdata, "kernel")
  File "/Users/paco/Documents/Projects/MobiLeak/Paper2-memory/memdump/sample-Rodeo2012/Volatility 2.3-devel/volatility/plugins/overlays/linux/linux.py", line 86, in parse_system_map
    (str_addr, symbol_type, symbol) = line.strip().split()
ValueError: too many values to unpack




On Fri, Feb 22, 2013 at 4:20 PM, Pasquale Stirparo <pstirparo@gmail.com> wrote:
Thanks Michael,

I had run
$ python vol.py --info
but obviously with all the info output I somehow did not see the different name of the htc profile... unbelievable but true :(

Anw thanks, it actually gives the output
Profiles
--------
...
LinuxHTCOneXx86        - A Profile for Linux HTCOneX x86

But now I'm getting other errors... each different for different plugins I'm trying :(
I'll try to understand where is the problem and come back to the list.

Thanks again



On Fri, Feb 22, 2013 at 3:59 PM, Michael Hale Ligh <michael.hale@gmail.com> wrote:
Hi Pasquale, 

Can you run this command:

# vol.py --info | grep Profile

The profile may not actually be named HTCOneX exactly. For example I have a Ubuntu1204.zip but the profile name is a little different: 

$ python vol.py --info | grep Profile
Volatile Systems Volatility Framework 2.3_alpha
....
LinuxUbuntu1204x64   - A Profile for Linux Ubuntu1204 x64
.....

So I would need to use --profile=LinuxUbuntu1204x64


On Wed, Feb 20, 2013 at 11:55 AM, Pasquale Stirparo <pstirparo@gmail.com> wrote:
Sorry guys... somehow I manage to push the keyboard shortcut and send the email before finishing. I'm sending it again compleet now.


Hi Andrew,

A summary hopefully useful also for others (especially mac users)
1) Thanks for the link related to dwarfdump, it worked indeed. It requires some dependancies, particularly for Mac OS X users libelf and elf.h (you can find it attached). This are the steps I followed:
- install libelf (you can download it from here http://freecode.com/projects/libelf)
- copy the file elf.h (attached) to /usr/include
- There is no make install, the make will compile the different tools. The developer suggest to use the new version .../dwarfdump2/dwarfdump. I'm also attaching the makefile sample updated to this.

2) Thanks. As I wrote in my previous email, I had find out that if exporting from the phone, on android you could find it under /proc/kallsyms :)

So, dwarfdump created a valid module.dwarf file and I did zip both it and kallsyms in HTCOneX.zip, and placed it inside the overlays/linux folder as follow (I've been trying using both volatility versions, the latest you can get from git and the 2.3_alpha from DFWRS):
hydra:linux paco$ zip /Users/paco/Documents/Projects/HTCOneX-vol-profile.zip module.dwarf /Users/paco/Documents/Tools/HTC-OneX/endeavoru-2.6.39-6b459dc/kallsyms   
  adding: module.dwarf (deflated 90%)
  adding: Users/paco/Documents/Tools/HTC-OneX/endeavoru-2.6.39-6b459dc/kallsyms (deflated 75%)

hydra:linux paco$ cp /Users/paco/Documents/Projects/HTCOneX-vol-profile.zip /Users/paco/Documents/Projects/sample-Rodeo2012/Volatility\ 2.3-devel/volatility/plugins/overlays/linux/HTCOneX.zip

hydra:linux paco$ cp /Users/paco/Documents/Projects/HTCOneX-vol-profile.zip /Users/paco/Documents/Tools/volatility/volatility-read-only/volatility/plugins/overlays/linux/HTCOneX.zip 

But when I run it I get still the error:
# python vol.py --profile=HTCOneX -f /Users/paco/Documents/Projects/memdump/20130213-2-htconex.lime linux_pslist
...
ERROR   : volatility.addrspace: Invalid profile HTCOneX selected


So chatting on the IRC channel with udgover, he suggested to have a look at volatility/plugins/overlays/linux.py file. In this file there is LinuxProfileFactory where it checks for "system.map" file. So I've tried to add the same routine for kallsyms. But it didn't work. I even tried to rename kallsyms to System.map and then zip it with module.dwarf. Still the same error.

I'll keep trying, if you have any clue in the meantime of the reason of the above error... here I am :)
Thanks guys

P.



On Wed, Feb 20, 2013 at 3:42 AM, Andrew Case <atcuno@gmail.com> wrote:
Hello,

Sorry for taking so long to respond...

There are a couple issues:

1) You need to use dwarfump from
http://reality.sgiweb.org/davea/dwarf.html , not the os x supplied
dwarfdump for Linux support.

2) You can get System.map values from a few places. First, check if
the source distributed for your kernel has it. Second, see if the
phone exports /proc/kallsyms. If neither of those work then we will
have to find another solution.

Thanks,
Andrew

On Sat, Feb 16, 2013 at 7:46 AM, Pasquale Stirparo <pstirparo@gmail.com> wrote:
> Thanks again Andrew for the assistance and for the Makefile. So here a brief
> summary for creating the Android profile, following the  linux example here
> (https://code.google.com/p/volatility/wiki/LinuxMemoryForensics):
>
> DISCLAIMER: I'm using a Mac OS X system.
>
> - I modified the Makefile sent by Andrew, placed under
> volatility/tools/linux/Makefile, adding the options for cross-compiling as
> from the LiME kernel module. Basically the modifications are the following:
>
> CCPATH :=
> /path/to/android-ndk/toolchains/arm-linux-androideabi-4.4.3/prebuilt/darwin-x86/bin
>
> dwarf: module.c
> $(MAKE) ARCH=arm CROSS_COMPILE=$(CCPATH)/arm-linux-androideabi- -C $(KDIR)
> CONFIG_DEBUG_INFO=y M=$(PWD) modules
>
>
> - The last instruction of the makefile "dwarfdump -di module.ko >
> module.dwarf" was originally with -di option. But -d appears to be not a
> valid option anymore. Checking the man page, the alternatives for debugging
> options are the following
> --debug-abbrev, --debug-aranges, --debug-frame[=OFFSET],
>       --debug-info[=OFFSET], --debug-inlined, --debug-line[=OFFSET],
>       --debug-macinfo[=OFFSET], --debug-pubnames[=PATTERN],
>       --debug-pubtypes[=PATTERN], --debug-str
> debug-info is the default (which looking at the output of "head
> module.dwarf" seems to be the correct one), so i tried just to remove the -d
> and also to put --debug-info without any offset value, but this is what I
> get as content of the module.dwarf file
> hydra:linux paco$ cat module.dwarf
> ----------------------------------------------------------------------
>  File: module.ko (arm)
> ----------------------------------------------------------------------
> .debug_info contents:
> < EMPTY >
>
> - The last step to make the profile is to zip the module.dwarf file and the
> System.map of the kernel. After searching around, I found that System.map
> file for android is /proc/kallsyms (probably you all already knew it, but I
> didn't know this :) ).
>
>
> Attached the make file I used, if all the rest of the steps I did are right,
> it's missing the right debug option for the dwarfdump command. I've tried
> also the others but no one seems to give the right output. Any tips on this?
> Thanks
>
> P.
>
>
>
> On Fri, Feb 15, 2013 at 5:41 PM, Andrew Case <atcuno@gmail.com> wrote:
>>
>> Hello,
>>
>> We are currently testing a stripped down Makefile to help people
>> compiling for different kernels than the one for the system they are
>> on (which includes Android). Can you please try the attached makefile?
>> You will need to change the KDIR varabile to point to your kernel
>> headers or source.
>>
>> On Fri, Feb 15, 2013 at 5:55 AM, Pasquale Stirparo <pstirparo@gmail.com>
>> wrote:
>> > Hi All,
>> >
>> > I'm trying to make a profile for android device.
>> > I did a memory dump with LiME of an HTC One X (Android 4.0.3, HTC Sense
>> > 4.0,
>> > kernel 2.6.39.4-g6b459dc).
>> >
>> > Now, following the instruction here
>> > https://code.google.com/p/volatility/wiki/LinuxMemoryForensics , I was
>> > trying to understand how to modify the makefile under
>> > volatility/tools/linux/ , in order to point to my kernel source. The
>> > thing
>> > is that in from my kernel source folder I couldn't find a proper value
>> > for
>> > KDIR and KVER (although they should be pretty straightforward according
>> > to
>> > their name) that would fit with the path for make command as from the
>> > following source code:
>> >
>> > pmem: pmem.c
>> >         $(MAKE) -C $(KDIR)/lib/modules/$(KVER)/build M=$(PWD) modules
>> >
>> > dwarf: module.c
>> >         $(MAKE) -C $(KDIR)/lib/modules/$(KVER)/build CONFIG_DEBUG_INFO=y
>> > M=$(PWD) modules
>> >         dwarfdump -di module.ko > module.dwarf
>> >         $(MAKE) -C $(KDIR)/lib/modules/$(KVER)/build M=$(PWD) clean
>> >
>> >
>> > Did anyone ever created an android profile? Any hint?
>> > I've seen in the mailing list archive a thread "Profile (ZIP) for
>> > Android
>> > 4.0.3" from Mike (in Cc), any news about that?
>> >
>> > Thank you
>> >
>> > P.
>> >
>> > --
>> > Pasquale Stirparo, MEng
>> > GCFA, OPST, OWSE, ECCE
>> >
>> > European Commission - JRC Joint Research Centre
>> > Institute for the Protection and Security of the Citizen (IPSC)
>> > Digital Citizen Security Unit
>> > Via E. Fermi, 2749 - TP 361
>> > 21027 Ispra (VA) - Italy
>> >
>> > PGP Key: 0x4C589FB2
>> > Fingerprint: 776D F072 3F43 D5DE CB55 86D2 55FF 14A7 4C58 9FB2
>> >
>> > Disclaimer: The views expressed are purely those of the writer and may
>> > not
>> > in any circumstance be regarded as stating an official position of the
>> > European Commission.
>> >
>> > _______________________________________________
>> > Vol-users mailing list
>> > Vol-users@volatilityfoundation.org
>> > http://lists.volatilityfoundation.org/mailman/listinfo/vol-users
>> >
>
>
>
>
> --
> Pasquale Stirparo, MEng
> GCFA, OPST, OWSE, ECCE
>
> European Commission - JRC Joint Research Centre
> Institute for the Protection and Security of the Citizen (IPSC)
> Digital Citizen Security Unit
> Via E. Fermi, 2749 - TP 361
> 21027 Ispra (VA) - Italy
>
> PGP Key: 0x4C589FB2
> Fingerprint: 776D F072 3F43 D5DE CB55 86D2 55FF 14A7 4C58 9FB2
>
> Disclaimer: The views expressed are purely those of the writer and may not
> in any circumstance be regarded as stating an official position of the
> European Commission.



--
Pasquale Stirparo, MEng    
GCFA, OPST, OWSE, ECCE                      

European Commission - JRC Joint Research Centre
Institute for the Protection and Security of the Citizen (IPSC)  
Digital Citizen Security Unit
Via E. Fermi, 2749 - TP 361
21027 Ispra (VA) - Italy

PGP Key: 0x4C589FB2
Fingerprint: 776D F072 3F43 D5DE CB55 86D2 55FF 14A7 4C58 9FB2

Disclaimer: The views expressed are purely those of the writer and may not in any circumstance be regarded as stating an official position of the European Commission.

_______________________________________________
Vol-users mailing list
Vol-users@volatilityfoundation.org
http://lists.volatilityfoundation.org/mailman/listinfo/vol-users





--
Pasquale Stirparo, MEng    
GCFA, OPST, OWSE, ECCE                      

European Commission - JRC Joint Research Centre
Institute for the Protection and Security of the Citizen (IPSC)  
Digital Citizen Security Unit
Via E. Fermi, 2749 - TP 361
21027 Ispra (VA) - Italy

PGP Key: 0x4C589FB2
Fingerprint: 776D F072 3F43 D5DE CB55 86D2 55FF 14A7 4C58 9FB2

Disclaimer: The views expressed are purely those of the writer and may not in any circumstance be regarded as stating an official position of the European Commission.



--
Pasquale Stirparo, MEng    
GCFA, OPST, OWSE, ECCE                      

European Commission - JRC Joint Research Centre
Institute for the Protection and Security of the Citizen (IPSC)  
Digital Citizen Security Unit
Via E. Fermi, 2749 - TP 361
21027 Ispra (VA) - Italy

PGP Key: 0x4C589FB2
Fingerprint: 776D F072 3F43 D5DE CB55 86D2 55FF 14A7 4C58 9FB2

Disclaimer: The views expressed are purely those of the writer and may not in any circumstance be regarded as stating an official position of the European Commission.




--
Pasquale Stirparo, MEng    
GCFA, OPST, OWSE, ECCE                      

European Commission - JRC Joint Research Centre
Institute for the Protection and Security of the Citizen (IPSC)  
Digital Citizen Security Unit
Via E. Fermi, 2749 - TP 361
21027 Ispra (VA) - Italy

PGP Key: 0x4C589FB2
Fingerprint: 776D F072 3F43 D5DE CB55 86D2 55FF 14A7 4C58 9FB2

Disclaimer: The views expressed are purely those of the writer and may not in any circumstance be regarded as stating an official position of the European Commission.