11:41 a.m.
By the way, I forgot Andrew has a copy of your CentOS memory dump. The
supplied patch (changing the shift) does work on your memory dump as well:
$ python vol.py --profile=Linuxnewcent2x64 -f /root/cent64/geriatrix.lime
linux_pslist
Volatile Systems Volatility Framework 2.3_alpha
Offset Name Pid Uid Gid
Start Time
------------------ -------------------- --------------- ---------------
------ ----------
0x00008100017407a0 init 1 0 0
Wed, 30 Jan 2013 12:21:20 +0000
0x0000810001740040 migration/0 2 0 0
Wed, 30 Jan 2013 12:21:20 +0000
0x00008100017437e0 ksoftirqd/0 3 0 0
Wed, 30 Jan 2013 12:21:20 +0000
0x0000810001743080 watchdog/0 4 0 0
Wed, 30 Jan 2013 12:21:20 +0000
On Mon, Feb 4, 2013 at 10:58 AM, Michael Hale Ligh
<michael.hale(a)gmail.com>wrote:
Hey Brian,
In a nutshell, we've investigated the issue and found that it's most
definitely not LiME causing the problem. LiME is producing a valid memory
dump. The problem is with volatility's DTB finding and address-space
verification checks (two different factors there). Fortunately, we believe
it to be a quite isolated and extraordinary case specific to CentOS 5.3 x64
with the 2.6.18.x kernel. I will explain a little about how we came to this
conclusion, however it will mostly be for documentation purposes. After
understanding the differences and considering the time involved in
supporting this "special" case, we probably won't be supporting it in the
future (also since its going on 4+ years old now).
The first task was actually finding install media for CentOS 5.3. I
tracked one down here:
http://www2.frugalware.org/mirror/centos.org/5.3/isos/x86_64/
After installing it, I built the volatility profile and dumped memory with
LiME. When trying to analyze with volatility, I got the same error as you
did, the main indicator was:
DEBUG : volatility.utils : Trying <class 'volatility.plugins.
addrspaces.amd64.AMD64PagedMemory'>
DEBUG1 : volatility.utils : Failed instantiating AMD64PagedMemory:
Failed valid Address Space check
The cause of this error message is the VolatilityLinuxIntelValidAS check
is failing. You can see the code here:
http://code.google.com/p/volatility/source/browse/trunk/volatility/plugins/….
To pass the check, the virtual address of the init_task structure, when
translated to a physical address, must equal the virtual address minus a
hard-coded "shift" value of 0xffffffff80000000 (which is consistent on all
x64 distributions and kernel versions that we've seen.
Before we can translate virtual addresses to physical, we need the kernel
DTB. This is found on typical systems by querying the init_level4_pgt
symbol from System.map and subtracting the shift value. As you can see,
this symbol exists in the CentOS 5.3 x64 2.6.18.x:
# grep init_level4_pgt System.map-2.6.18-128.el5
ffffffff80001000 T init_level4_pgt
When you subtract the shift, it leaves 0x1000. If you view this offset in
the *raw* memory dump (not the lime formatted memory dump, because that has
headers which will change the offset), you'll see that 0x1000 is most
definitely not a DTB, its all zeros:
$ xxd -s 0x1000 ~/Documents/Virtual\
Machines.localized/CentOS53x64.vmwarevm/CentOS53x64-Snapshot1.vmem | less
0001000: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0001010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0001020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0001030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0001040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0001050: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0001060: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0001070: 0000 0000 0000 0000 0000 0000 0000 0000 ................
After digging through the kernel source, Andrew found that this particular
kernel uses a symbol called boot_level4_pgt to store the kernel DTB
instead. But:
# grep boot_level4_pgt System.map-2.6.18-128.el5
#
So, what we have is a System.map that initializes init_level4_pgt to a
value that isn't the DTB, and the value that stores the real DTB is not in
the map file. We've never heard of boot_level4_pgt, so apparently it was a
very short-lived thing.
Anyway, back on the live CentOS 5.3 system, we loaded an lkm that read and
printed the value of the init_task structure's mm->pgd value. It came up
with 0x3e58d000 for my VM. I went back and plugged that into the --dtb
parameter to volatility (which will override the value it gets from
init_level4_pgt):
$ python2.6 vol.py -f ~/Desktop/centos53x64-128.lime
--profile="LinuxCentOS53-128x64" --dtb=0x3e58d000
There was still an issue with verifying the AMD64PagedMemory address
space, however this time the virtual address of the init_task was actually
translated properly (since we supplied the DTB manually). Some debug
messaged showed that the check was failing because the shift value was
0x200000 bytes off. So a very simple patch of the following nature was able
to fix everything up:
Index: volatility/plugins/overlays/linux/linux64.py
===================================================================
--- volatility/plugins/overlays/linux/linux64.py (revision 3020)
+++ volatility/plugins/overlays/linux/linux64.py (working copy)
@@ -33,7 +33,7 @@
"""Tries to locate the DTB."""
profile = self.obj_vm.profile
- yield profile.get_symbol("init_level4_pgt") - 0xffffffff80000000
+ yield profile.get_symbol("init_level4_pgt") - 0xffffffff7fe00000
class Linux64ObjectClasses(obj.ProfileModification):
Index: volatility/plugins/overlays/linux/linux.py
===================================================================
--- volatility/plugins/overlays/linux/linux.py (revision 3020)
+++ volatility/plugins/overlays/linux/linux.py (working copy)
@@ -656,7 +656,7 @@
if self.obj_vm.profile.metadata.get('memory_model', '32bit') ==
"32bit":
shift = 0xc0000000
else:
- shift = 0xffffffff80000000
+ shift = 0xffffffff7fe00000
yield self.obj_vm.vtop(init_task_addr) == init_task_addr - shift
Now we can find the DTB and pass the AMD64 address space check with no
issues:
$ python2.6 vol.py -f ~/Desktop/centos53x64-128.lime
--profile="LinuxCentOS53-128x64" linux_pslist
Volatile Systems Volatility Framework 2.3_alpha
Offset Name Pid Uid
Gid DTB Start Time
------------------ -------------------- --------------- ---------------
------ ------------------ ----------
......
0x000081003cbbf080 firefox 6905 500
500 0x000000002256a000 Sun, 03 Feb 2013 12:19:45 +0000
0x000081001bf08860 su 7008 500
500 0x0000000018130000 Sun, 03 Feb 2013 12:21:04 +0000
0x000081002d899100 bash 7011 0 0
0x0000000025b19000 Sun, 03 Feb 2013 12:21:05 +0000
0x000081002a3bf820 notification-da 9374 500
500 0x000000001f444000 Sun, 03 Feb 2013 12:26:40 +0000
0x000081003fe6e7a0 bash 10392 500
500 0x0000000014785000 Sun, 03 Feb 2013 12:34:50 +0000
0x0000810014f9a100 insmod 10597 0 0
0x0000000012399000 Sun, 03 Feb 2013 12:36:11 +0000
$ python2.6 vol.py -f ~/Desktop/centos53x64-128.lime
--profile="LinuxCentOS53-128x64" linux_dmesg
Volatile Systems Volatility Framework 2.3_alpha
WARNING : volatility.obj : Overlay structure cpuinfo_x86 not present
in vtypes
<5>Linux version 2.6.18-128.el5 (mockbuild(a)builder10.centos.org) (gcc
version 4.1.2 20080704 (Red Hat 4.1.2-44)) #1 SMP Wed Jan 21 10:41:14 EST
2009
<6>Command line: ro root=LABEL=/ quiet
<6>BIOS-provided physical RAM map:
<4> BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
<4> BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
<4> BIOS-e820: 00000000000ca000 - 00000000000cc000 (reserved)
$ python2.6 vol.py -f ~/Desktop/centos53x64-128.lime
--profile="LinuxCentOS53-128x64" linux_ifconfig
Volatile Systems Volatility Framework 2.3_alpha
WARNING : volatility.obj : Overlay structure cpuinfo_x86 not present
in vtypes
Interface IP Address MAC Address Promiscous Mode
---------------- -------------------- ------------------ ---------------
lo 127.0.0.1 00:00:00:00:00:00 False
eth0 192.168.16.240 00:0c:29:d4:30:9e False
sit0 0.0.0.0 00:00:00:00:00:00 False
One of the reasons the shift *could* be different is if CONFIG_RELOCATABLE
kernel option is set. However, on my CentOS 5.3 x64 2.6.18.x, the option is
*not* enabled and I'm guessing its not on your either. Plus, Andrew has
analyzed systems in the past with this option *enabled* yet the shift value
was still 0xffffffff80000000.
So in the future if this becomes a problem, we may consider some
fuzzing/brute-forcing around the 0xffffffff80000000 plus or minus 0x200000
area, but currently its only been seen as different on on the 4+ year old
CentOS 5.3 x64 2.6.18.x, so its not a major issue for us at this time.
Hope this helps!
MHL
On Thu, Jan 31, 2013 at 11:24 AM, Brian Keefer <chort(a)effu.se> wrote:
Hello, sending to list for archive(?) sake.
chort@hydra:~/code/profiles-volatility/CentOS_5.3_x64$ vol.py
--profile=LinuxCentOS_5_3x64 -f /fun/ir/geriatrix.lime -dd linux_lsmod
Volatile Systems Volatility Framework 2.3_alpha
DEBUG : volatility.utils : Voting round
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.macho.MachOAddressSpace'>
DEBUG1 : volatility.utils : Failed instantiating MachOAddressSpace:
mac: need base
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.lime.LimeAddressSpace'>
DEBUG1 : volatility.utils : Failed instantiating LimeAddressSpace:
lime: need base
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.hibernate.WindowsHiberFileSpace32'>
DEBUG1 : volatility.utils : Failed instantiating
WindowsHiberFileSpace32: No base Address Space
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace64'>
DEBUG1 : volatility.utils : Failed instantiating
WindowsCrashDumpSpace64: No base Address Space
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.hpak.HPAKAddressSpace'>
DEBUG1 : volatility.utils : Failed instantiating HPAKAddressSpace: No
base Address Space
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.vboxelf.VirtualBoxCoreDumpElf64'>
DEBUG1 : volatility.utils : Failed instantiating
VirtualBoxCoreDumpElf64: No base Address Space
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.vmware.VMWareSnapshotFile'>
DEBUG1 : volatility.utils : Failed instantiating VMWareSnapshotFile:
No base Address Space
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace32'>
DEBUG1 : volatility.utils : Failed instantiating
WindowsCrashDumpSpace32: No base Address Space
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.intel.JKIA32PagedMemoryPae'>
DEBUG1 : volatility.utils : Failed instantiating
JKIA32PagedMemoryPae: No base Address Space
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.amd64.AMD64PagedMemory'>
DEBUG1 : volatility.utils : Failed instantiating AMD64PagedMemory: No
base Address Space
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.intel.JKIA32PagedMemory'>
DEBUG1 : volatility.utils : Failed instantiating JKIA32PagedMemory:
No base Address Space
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.legacyintel.IA32PagedMemoryPae'>
DEBUG1 : volatility.utils : Failed instantiating IA32PagedMemoryPae:
Module disabled
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.legacyintel.IA32PagedMemory'>
DEBUG1 : volatility.utils : Failed instantiating IA32PagedMemory:
Module disabled
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.standard.FileAddressSpace'>
DEBUG : volatility.plugins.overlays.linux.linux: CentOS_5.3: Found
dwarf file System.map-2.6.18-128.el5 with 365 symbols
DEBUG : volatility.plugins.overlays.linux.linux: CentOS_5.3: Found
system file System.map-2.6.18-128.el5 with 1 symbols
DEBUG : volatility.obj : Applying modification from BashTypes
DEBUG : volatility.obj : Applying modification from
BasicObjectClasses
DEBUG : volatility.obj : Applying modification from
ELF64Modification
DEBUG : volatility.obj : Applying modification from HPAKVTypes
DEBUG : volatility.obj : Applying modification from LimeTypes
DEBUG : volatility.obj : Applying modification from MachoTypes
DEBUG : volatility.obj : Applying modification from MbrObjectTypes
DEBUG : volatility.obj : Applying modification from
VMwareVTypesModification
DEBUG : volatility.obj : Applying modification from
VirtualBoxModification
DEBUG : volatility.obj : Applying modification from
LinuxKmemCacheOverlay
DEBUG : volatility.obj : Applying modification from
LinuxMountOverlay
DEBUG : volatility.obj : Applying modification from
LinuxObjectClasses
DEBUG : volatility.obj : Applying modification from LinuxOverlay
WARNING : volatility.obj : Overlay structure cpuinfo_x86 not present
in vtypes
DEBUG : volatility.obj : Applying modification from
Linux64ObjectClasses
DEBUG : volatility.utils : Succeeded instantiating
<volatility.plugins.addrspaces.standard.FileAddressSpace object at
0x5c93d50>
DEBUG : volatility.utils : Voting round
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.macho.MachOAddressSpace'>
DEBUG1 : volatility.utils : Failed instantiating MachOAddressSpace:
MachO Header signature invalid
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.lime.LimeAddressSpace'>
DEBUG1 : volatility.obj : None object instantiated: Invalid Address
0x3FF8F860, instantiating lime_header
DEBUG : volatility.utils : Succeeded instantiating
<volatility.plugins.addrspaces.lime.LimeAddressSpace object at 0x5c93d10>
DEBUG : volatility.utils : Voting round
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.macho.MachOAddressSpace'>
DEBUG1 : volatility.utils : Failed instantiating MachOAddressSpace:
MachO Header signature invalid
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.lime.LimeAddressSpace'>
DEBUG1 : volatility.utils : Failed instantiating LimeAddressSpace:
Invalid Lime header signature
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.hibernate.WindowsHiberFileSpace32'>
DEBUG1 : volatility.utils : Failed instantiating
WindowsHiberFileSpace32: PO_MEMORY_IMAGE is not available in profile
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace64'>
DEBUG1 : volatility.utils : Failed instantiating
WindowsCrashDumpSpace64: Header signature invalid
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.hpak.HPAKAddressSpace'>
DEBUG1 : volatility.utils : Failed instantiating HPAKAddressSpace:
Invalid magic found
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.vboxelf.VirtualBoxCoreDumpElf64'>
DEBUG1 : volatility.utils : Failed instantiating
VirtualBoxCoreDumpElf64: ELF64 Header signature invalid
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.vmware.VMWareSnapshotFile'>
DEBUG1 : volatility.utils : Failed instantiating VMWareSnapshotFile:
Invalid VMware signature: 0xf000ff53
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.crash.WindowsCrashDumpSpace32'>
DEBUG1 : volatility.utils : Failed instantiating
WindowsCrashDumpSpace32: Header signature invalid
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.intel.JKIA32PagedMemoryPae'>
DEBUG1 : volatility.utils : Failed instantiating
JKIA32PagedMemoryPae: Incompatible profile LinuxCentOS_5_3x64 selected
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.amd64.AMD64PagedMemory'>
DEBUG1 : volatility.utils : Failed instantiating AMD64PagedMemory:
Failed valid Address Space check
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.intel.JKIA32PagedMemory'>
DEBUG1 : volatility.utils : Failed instantiating JKIA32PagedMemory:
Incompatible profile LinuxCentOS_5_3x64 selected
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.legacyintel.IA32PagedMemoryPae'>
DEBUG1 : volatility.utils : Failed instantiating IA32PagedMemoryPae:
Module disabled
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.legacyintel.IA32PagedMemory'>
DEBUG1 : volatility.utils : Failed instantiating IA32PagedMemory:
Module disabled
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.standard.FileAddressSpace'>
DEBUG1 : volatility.utils : Failed instantiating FileAddressSpace:
Must be first Address Space
DEBUG : volatility.utils : Trying <class
'volatility.plugins.addrspaces.arm.ArmAddressSpace'>
DEBUG1 : volatility.utils : Failed instantiating ArmAddressSpace:
Incompatible profile LinuxCentOS_5_3x64 selected
No suitable address space mapping found
Tried to open image as:
MachOAddressSpace: mac: need base
LimeAddressSpace: lime: need base
WindowsHiberFileSpace32: No base Address Space
WindowsCrashDumpSpace64: No base Address Space
HPAKAddressSpace: No base Address Space
VirtualBoxCoreDumpElf64: No base Address Space
VMWareSnapshotFile: No base Address Space
WindowsCrashDumpSpace32: No base Address Space
JKIA32PagedMemoryPae: No base Address Space
AMD64PagedMemory: No base Address Space
JKIA32PagedMemory: No base Address Space
IA32PagedMemoryPae: Module disabled
IA32PagedMemory: Module disabled
MachOAddressSpace: MachO Header signature invalid
MachOAddressSpace: MachO Header signature invalid
LimeAddressSpace: Invalid Lime header signature
WindowsHiberFileSpace32: PO_MEMORY_IMAGE is not available in profile
WindowsCrashDumpSpace64: Header signature invalid
HPAKAddressSpace: Invalid magic found
VirtualBoxCoreDumpElf64: ELF64 Header signature invalid
VMWareSnapshotFile: Invalid VMware signature: 0xf000ff53
WindowsCrashDumpSpace32: Header signature invalid
JKIA32PagedMemoryPae: Incompatible profile LinuxCentOS_5_3x64 selected
AMD64PagedMemory: Failed valid Address Space check
JKIA32PagedMemory: Incompatible profile LinuxCentOS_5_3x64 selected
IA32PagedMemoryPae: Module disabled
IA32PagedMemory: Module disabled
FileAddressSpace: Must be first Address Space
ArmAddressSpace: Incompatible profile LinuxCentOS_5_3x64 selected
--
chort
On Jan 31, 2013, at 8:20 AM, Andrew Case wrote:
Hey,
Can you run again with "-dd" added before linux_lsmod and send me the
output?
The lack of cpuinfo_x86 does not change lsmod so do not worry about
that part...
On Thu, Jan 31, 2013 at 1:18 AM, Brian Keefer <chort(a)effu.se> wrote:
> I built LiME from the tarball on the project site (not latest svn) and
was
able to dump memory successfully (type=lime). After many trials and
tribulations I was able to get the Volatility profile built for CentOS
5.3x64 (had to remove pmem from the Makefile). I put the profile in the
correct directory, and vol.py --info lists it as expected, however when I
try to use the profile with the memory image I get an error.
>
> chort@hydra:~/code/profiles-volatility/CentOS_5.3_x64$ vol.py
--profile=LinuxCentOS_5_3x64 -f /fun/ir/geriatrix.lime linux_lsmod
> Volatile Systems Volatility Framework
2.3_alpha
> WARNING : volatility.obj : Overlay structure cpuinfo_x86 not
present in
vtypes
> No suitable address space mapping found
> Tried to open image as:
> MachOAddressSpace: mac: need base
> LimeAddressSpace: lime: need base
> WindowsHiberFileSpace32: No base Address Space
> WindowsCrashDumpSpace64: No base Address Space
> HPAKAddressSpace: No base Address Space
> VirtualBoxCoreDumpElf64: No base Address Space
> VMWareSnapshotFile: No base Address Space
> WindowsCrashDumpSpace32: No base Address Space
> JKIA32PagedMemoryPae: No base Address Space
> AMD64PagedMemory: No base Address Space
> JKIA32PagedMemory: No base Address Space
> IA32PagedMemoryPae: Module disabled
> IA32PagedMemory: Module disabled
> MachOAddressSpace: MachO Header signature invalid
> MachOAddressSpace: MachO Header signature invalid
> LimeAddressSpace: Invalid Lime header signature
> WindowsHiberFileSpace32: PO_MEMORY_IMAGE is not available in profile
> WindowsCrashDumpSpace64: Header signature invalid
> HPAKAddressSpace: Invalid magic found
> VirtualBoxCoreDumpElf64: ELF64 Header signature invalid
> VMWareSnapshotFile: Invalid VMware signature: 0xf000ff53
> WindowsCrashDumpSpace32: Header signature invalid
> JKIA32PagedMemoryPae: Incompatible profile LinuxCentOS_5_3x64 selected
> AMD64PagedMemory: Failed valid Address Space check
> JKIA32PagedMemory: Incompatible profile LinuxCentOS_5_3x64 selected
> IA32PagedMemoryPae: Module disabled
> IA32PagedMemory: Module disabled
> FileAddressSpace: Must be first Address Space
> ArmAddressSpace: Incompatible profile LinuxCentOS_5_3x64 selected
>
> On a hunch I checked the directory I built the profile in (copied
headers
& source from the target system):
>
chort@hydra:~/code/profiles-volatility/CentOS_5.3_x64$ grep cpuinfo *
> System.map-2.6.18-128.el5:ffffffff8006f328 t show_cpuinfo
> System.map-2.6.18-128.el5:ffffffff80103251 t cpuinfo_open
> System.map-2.6.18-128.el5:ffffffff8020eadb t show_cpuinfo_max_freq
> System.map-2.6.18-128.el5:ffffffff8020eafa t show_cpuinfo_min_freq
> System.map-2.6.18-128.el5:ffffffff8020f759 t show_cpuinfo_cur_freq
> System.map-2.6.18-128.el5:ffffffff802f0bc0 D cpuinfo_op
> System.map-2.6.18-128.el5:ffffffff80308420 d proc_cpuinfo_operations
> System.map-2.6.18-128.el5:ffffffff803319a0 d cpuinfo_cur_freq
> System.map-2.6.18-128.el5:ffffffff80331b20 d cpuinfo_min_freq
> System.map-2.6.18-128.el5:ffffffff80331b60 d cpuinfo_max_freq
>
>
> Platform running Volatility (2.3_alpha, latest from svn):
> Linux hydra 3.2.0-35-generic #55-Ubuntu SMP Wed Dec 5 17:42:16 UTC
2012 x86_64
x86_64 x86_64 GNU/Linux
>
> Source of memory image:
> Linux geriatrix.smtps.net 2.6.18-128.el5 #1 SMP Wed Jan 21 10:41:14
EST 2009
x86_64 x86_64 x86_64 GNU/Linux
>
> What am I missing?
>
>
> --
> chort
>
>
>
> _______________________________________________
> Vol-users mailing list
> Vol-users(a)volatilityfoundation.org
> http://lists.volatilityfoundation.org/mailman/listinfo/vol-users
_______________________________________________
Vol-users mailing list
Vol-users(a)volatilityfoundation.org
http://lists.volatilityfoundation.org/mailman/listinfo/vol-users