8:42 p.m.
Hi George, and everyone else on the list.
I don't know if you watched/skimmed through the presentation or the talk,
but I want to emphasize that I really like Volatility and that my work is
not a research against Volatility (or any other tool in particular) in any
way.
Volatility is my primary choice when doing memory analysis both because of
speed, abundance of features and plugins and openness of the tool.
I do however mention Volatility in couple of places, for example, I suggest
creating a plugin for scanning handle table allocations (Obtb) as an
alternative to currently used method.
It is nice to see someone looking critically at the
subject matter, even
if in an over-simplistic manner. The gist of the article is that you
can easily scrub evidence from a memory dump as it is being written (in
plain text) to disk or to the net. Duh!
Thanks for your support.
Although I agree some things were over-simplified, I don't agree that my
research in whole is over-simplistic.
The idea is simple and well known enough, still, it is not that commonly
being taken into consideration when performing forensic analysis, at least
from my experience.
A few comments on the author's conclusions:
Too bad you pulled out the conclusions since this slide is by far the
weakest in the whole presentation;)
1. Acquisition tools should utilize drivers correctly!
Duh!
I didn't quite understand if you meant that ironically or not. However,
almost all tested acquisition tools utilize drivers incorrectly, meaning
that they transfer the pages back to UM and then write them to dump (usually
with completely inadequate permissions).
2. Use hardware acquisition tools, e.g. firewire.
However, hardware-based acquisition also can be
defeated. At a minimum
you can program an upper limit on the memory address that the firewire
is allowed to access and then place your rootkit above that address.
Most new computer systems have more than 4 GiB of memory nowadays.
I didn't say that hardware acquisition methods are infallible. You're
absolutely right regarding the 4 GB limit. But from an attacker's
perspective, this method cannot be used for hiding arbitrary object, and it
might be difficult to "relocate" the driver, allocations and all resources
above the specified limit.
I'm evaluating some additional weaknesses of the firewire acquisition and
methods of hiding arbitrary objects, but so far, my research showed no
progress...
And yes, I am definitely aware of other alternatives (cold boot etc.), but I
think they are highly impractical so I didn't mention them in my
presentation.
3. Use crash dumps (native!) instead of raw dumps.
Should maybe introduce the author to all those
rootkits (e.g. Sinowal)
that remove themselves from crashdump as it is being written.
I thought that I have a very good knowledge of rootkits and low level
malware, but I don't recall Torpig/Sinowal doing anything about removing
itself from the crash-dump.
In fact, I cannot remember any rootkit or malware removing its artifacts
from the crash-dump, so please introduce me to all such rootkits:)
There are some KeBugCheckEx() tampering "malware" variants, but they are
mostly related to PatchGuard bypass.
Crash dump mechanism is fairly complex, involves couple of checksums and
does not use the underlying file system - it writes the dump to raw disk
sectors.
The weakest link in this process from my perspective is the process of
booting the machine and calling of SmpCheckForCrashDump() function - this is
the point when malware can finally see the resulting crash dump and remove
the artifacts from it.
This is far from being trivial, however.
4. Perform anti-rootkit scanning before acquisition?
Easier said than done. Just ask A/V industry.
This is a stupid conclusion on my side. No, really, I said that during the
talk:)
5. Live forensic is inherently insecure!
Duh! Real question is not whether or not you can
cheat memory
acquisition software. It is whether you can cheat memory acquisition
software and have no one know about it. Knowing that a system is
infected is 90% of the battle even if you don't know how. Once I know
that a system is infected I will find the rootkit.
My talk was all about cheating the memory acquisition software and have no
one know about it:)
It's not that my tool is invisible - the truth is actually totally opposite
ON PURPOSE.
I didn't want to add advanced self-hiding features, because I'm not trying
to create a tool that bad guys can use in the wild. I just wanted to raise
awareness and present my research.
And again, it depends on what you are trying to find:) If you're trying to
find _traces_ of malware (mostly orphaned or invalid resources, such as
locks, mutants etc.), you will most certainly succeed in this, because by
theory it is impossible to hide everything (the malware executes, therefore
it is present and visible).
If your goal is to find some more valuable artifacts (processes, memory
allocations, files, etc.), you might fail even if you are certain in rootkit
existance, simply because the rootkit (or my tool in this case) has deleted
such artifacts from the dump.
In the end, I really don't want to be perceived as a guy wearing a wrong
hat. My research is not aimed against any particular product, or the memory
forensics field in general. My research is not a revolutionary new
anti-forensic method. It's an implementation of well known anti-forensic
techniques and proof of concept for removing arbitrary artifacts from the
memory dump with the intention of raising the awareness and creating better
memory forensics software.
Best regards,
Luka
--
Luka Milković
Information security consultant
e-mail: luka.milkovic(a)infigo.hr
INFIGO IS d.o.o.
Horvatovac 20, 10000 Zagreb
www: http://www.infigo.hr
10:10 a.m.
Luka,
Good to hear from you and thanks again for drawing attention to the
importance of reliability issues in volatile evidence acquisition. The
robustness and reliability of memory acquisition tools has received
scant attention up until now, Yet, the acquisition phase is the most
important step. You can run an analysis tool and, if it fails, you can
still run a different analysis tool or patch the first analysis tool and
run it again. If an acquisition tool fails silently you are typically
out of luck! Note that the reliability issues addressed in your
presentation are not unique to memory acquisition but also apply to the
live acquisition of other storage media as well. Darren Bilby's 2006
Blackhat presentation (DDefy) addressed both "live" memory and other
storage acquisition. While your current presentation focuses on
physical memory acquisition, your techniques could easily be applied to
the "live" acquisition of fixed storage media as well. And perhaps they
should be so applied.
My apologies if my previous post seemed dismissive. The appropriate
response to a critical work is to view it critically, however. Darren
unfortunately abandoned his work on DDefy. My intention is not that you
abandon Dementia but that you improve it! I wouldn't worry too much
about giving the "bad guys" ideas, as they already are implementing
plenty of ideas, including some that haven't been mentioned yet.
Paper documents can be forged. Electronic documents can be tampered
with. Volatile evidence can be modified. We know that evidence can be
falsified. The question is, "What should we do about it?" That is why
the last slide in your presentation is the most important. Because it
attempts to provide us with a road map for the future, a way forward.
Unfortunately that is where I am having my greatest difficulty, as I
will explain.
1. Acquisition tools should utilize drivers correctly, (i.e. they should
acquire memory into a kernel mode buffer and write that buffer to a file
from kernel mode)!
It is true that a tool such as win32dd will not be susceptible to user
mode DeviceIoControl/NtDeviceIoControlFile or WriteFile/NtWriteFile
hooks. However, win32dd will be/is vulnerable to your kernel mode
NtWriteFile hook and to a file system filter driver (assuming that you
are implementing those methods correctly). Also, if you are able to
install an inline hook in the kernel mode NtWriteFile function, you
ought equally to be able to install inline hooks in NtMapViewOfSection,
MmMapIoSpace and MmMapMemoryDumpMdl. Or the same could be accomplished
using IAT hooks, like the driver verifier does it. Indeed, hooking
those three functions would cover all of the memory acquisition tools
which you tested and render output compression or encryption irrelevant.
So how is the KM/KM paradigm (e.g. win32dd) superior to other methods?
Your PoC itself subverts both UM and KM acqusition, assuming that you
have implemented your NtWriteFile hook correctly. If not then shame on
you! Note that I am not saying this to disparage win32dd. I am just
saying that the approach taken by this tool is not inherently more
reliable than other methods.
2. Use hardware acquisition tools.
You're absolutely right regarding the 4 GB limit.
But from
an attacker's perspective, this method cannot be used for hiding
arbitrary object, and it might be difficult to "relocate" the
driver, allocations and all resources above the specified limit. <
Actually it is quite easy to load a driver or arbitrary object (e.g.
encryption key or key schedules) into a reserved physical address space
above 4 GiB. We are currently shipping some software to our customers
that is able to do precisely that. It wasn't written with the intention
of "cheating" firewire memory dumps; however, it should serve that
purpose equally well.
3. Use crash dumps (native!) instead of raw dumps.
I may have been mistaken about Sinowal removing itself from crashdumps
during the dump process. Of the publicly available rootkits, Rustock.C
appears to be the one most often cited as an example of this.
http://go.eset.com/us/resources/white-papers/Yet_Another_Rustock_Analysis.p…;
http://www.eicar.org/files/lipovsky_eicar2010.pdf. A very cursory
static analysis of crashdmp!WriteBufferToDisk indicates that
BugCheckDumpIoCallback's are called before data is written to the dump file.
uf crashdmp!WriteBufferToDisk
[...]
fffff880`00ac450a ff15f85b0000 call qword ptr
[crashdmp!_imp_MmBuildMdlForNonPagedPool (fffff880`00aca108)]
fffff880`00ac4510 448b4c2428 mov r9d,dword ptr [rsp+28h]
fffff880`00ac4515 448b442440 mov r8d,dword ptr [rsp+40h]
fffff880`00ac451a 488b542470 mov rdx,qword ptr [rsp+70h]
fffff880`00ac451f b801000000 mov eax,1
fffff880`00ac4524 488bce mov rcx,rsi
fffff880`00ac4527 660944245a or word ptr [rsp+5Ah],ax
fffff880`00ac452c e87f0e0000 call crashdmp!InvokeDumpCallbacks
(fffff880`00ac53b0)
and then:
uf crashdmp!InvokeDumpCallbacks
[...]
fffff880`00ac5442 488364242800 and qword ptr [rsp+28h],0
fffff880`00ac5448 33c0 xor eax,eax
fffff880`00ac544a 4889442430 mov qword ptr [rsp+30h],rax
fffff880`00ac544f 4889442438 mov qword ptr [rsp+38h],rax
fffff880`00ac5454 48834c2428ff or qword ptr
[rsp+28h],0FFFFFFFFFFFFFFFFh
fffff880`00ac545a 4c897c2430 mov qword ptr [rsp+30h],r15
fffff880`00ac545f 4489742438 mov dword ptr [rsp+38h],r14d
fffff880`00ac5464 8974243c mov dword ptr [rsp+3Ch],esi
fffff880`00ac5468 448d4818 lea r9d,[rax+18h] r9d =
sizeof(KBUGCHECK_DUMP_IO)
fffff880`00ac546c 4c8d442428 lea r8,[rsp+28h] ; r8 =
KBugcheckDumpIoBlock
fffff880`00ac5471 488bd3 mov rdx,rbx ; rdx = CallbackRecord
fffff880`00ac5474 8d4803 lea ecx,[rax+3]; ecx =
KbCallbackDumpIo
fffff880`00ac5477 ff5310 call qword ptr [rbx+10h]
The dump IO block contains a pointer to the data that is about to be
written to the dump file. Even if it turns out that this callback is
being called after data has been written to the dump file you could use
the first call (ReasonSpecificData->Type == KbDumpIoHeader) to remove
the rootkit from memory and then ignore subsequent calls.
The bigest problem with crash dumps is that the dump process discards so
much valuable evidence. Basically, you only get what is in the OS's
physical memory map with a crashdump. That will miss the real mode
interrupt vector table, many artifacts of ACPI rootkits and BIOS kits,
and rootkits loaded into unmanaged memory. In addition, crashdumps lack
some of the features that we expect from a trusted forensic tool such as
cryptographic checksums and robust error logging. Finally, the
crashdump overwrites the pagefile which itself may contain valuable
evidence.
Once again, the superiority of crashdumps over other acquisition methods
appears to be largely due to a design decision on your part not to
attack crashdumps as vigorously as you might.
So where does that leave us? All hope is lost?
Paper documents can be forged. Electronic documents can be tampered
with. Volatile evidence can be modified. Computer forensics scarcely
could go on if the mere possibility or even fact of evidence tampering
were sufficient to invalidate the whole process. If evidence were
inherently reliable we would not need experts to acquire it or to apply
a forensic method to sift the facts. One of the major limitations of
your presentation is that it does not address the methods that
professional forensic tools use (or should use) to ensure the integrity
of evidence as it is being transmitted and archived for future use, e.g.
robust logging and cryptographic checksums. To be successful the "bad
guy" must not only alter volatile evidence, he must also keep the
investigator from knowing that the evidence has been altered. "Bad
guys" try to "hide" or alter evidence in all sorts of different ways.
But as Jesse Kornblum has pointed out, the act of "hiding" itself
produces artifacts which enhance the visibility of the malware that is
being hid (this is the "rootkit paradox"). Before we begin to ask about
malicious modification of volatile evidence an even more basic question
needs to be asked: Does the memory acquisition tool implement a sound
forensic method in the first place? If the tool does not offer any
means to ensure evidentiary integrity (e.g. robust logging and
cryptographic checksums) there is no need to ask about malicious
modification of evidence.
Once we are satisfied that a tool implements a sound forensic method we
would then like to see if the evidence adulteration is reflected in the
output of the tool in the form of log entries and/or modified
cryptographic checksum. In other words, I would like to see you acquire
memory evidence the way that a professional computer forensic "expert"
would acquire evidence and then see whether the PoC is able to subvert
the acquisition process without any indication in the form of log
entries or altered cryptographic checksums.
All of the techniques which you document, except for the user mode
DeviceIoControl hook, would be defeated simply by adopting a
forensically sound method. If any thing I draw the conclusion that a
crytographic hash of the data should be generated very early in the
acquisition process, and as close to the actual reading of the data as
possible. If you were to conclude that memory acquisition tools should
generate cryptographic hashes of the evidence in KM I would buy that!
Cryptographic checksums might still be defeated by NtMapViewOfSection,
MmMapIoSpace and MmMapMemoryDumpMdl hooks. But we can detect their
operation using other means.
While not strictly a part of the acquisition process, robust analysis is
also a means to ensure evidentiary reliability. Even when robust
logging and cryptographic hashes fail, robust analysis still may be able
to detect anomalies which indicate evidence tampering. To draw a couple
of analogies from the world of disk forensics, "bad guys" often tamper
with file system MAC times. But then forensic investigators look for
artifacts of altered MAC times (or the tools which generated them)
elsewhere in the file system image. "Bad guys" also use file or disk
wiping software (e.g. CCleaner) to destroy digital evidence. Forensic
experts then look for artifacts that CCleaner was used on the system
within the time frame that evidence is believed to have been destroyed.
How is using Dimentia to wipe certain artifacts from memory different
from using CCleaner to destroy disk evidence, if I can find artifacts
which show that Dimentia was used? Inline hooks in a user mode process
leave some definite artifacts even after they have been removed. Kernel
mode inline hooks (e.g. in NtWriteFile) also are highly visible, and
they don't work on 64-bit Windows as you acknowledge. Even the lower
disk filters used by most bootkits are visible now that everyone knows
to look for them.
In our own research we have not had much trouble coming up with ways to
"cheat" physical memory acquisition. What is difficult is finding ways
to "cheat" the acquisition process without anyone knowing about it
(assuming that a sound forensic method was applied during acquisition).
To be successful anti-forensic software must accomplish its purpose
while leaving very few traces behind. One approach may be to place the
rootkit in a memory location that the forensic tool does not acquire.
For example, some tools aren't able to acquire non-cached or
write-combined memory on Windows systems prior to Windows 7 (e.g. MDD).
You might be able to "hide" a rootkit from one of these tools simply
by placing the rootkit in non-cached or write-combined memory without
any hooks whatsoever. (Well, actually, only the appropriate entry in
the PFN database needs to say that the memory is non-cached or
write-combined, and not the PTE entry itself.)
Robust logging and proper use of cryptographic checksums are an
essential attribute of a trusted forensic tool. If your PoC were to
force forensic tool vendors to implement sound forensic methods, or
failing that, to show the perils of not implementing those methods, it
would do a great service to the forensic community. To be fair you
should not ignore suitable and obvious methods of subverting KM
acquisition methods. (Why is it that you chose not to attack
NtMapViewOfSection, MmMapIoSpace and MmMapMemoryDumpMdl directly?) A
very interesting extension of your work would be to modify open source
router firmware to strip malicious artifacts from the data stream as it
is being transmitted over the wire. Then see how many tools notice that
the evidence was modified. :-)
Just for the record, we strongly urge our customers to acquire evidence
to the network when the presence of malware is suspected. Malware that
spreads via removable storage devices is quite common now-a-days.
Evidence acquired to a local storage device needs special handling to
avoid spreading the infection to other computers on site and to avoid
the investigator becoming part of someone's botnet. We also strongly
recommend the use of content encryption (and not just channel
encryption) when acquiring over the network to prevent the disclosure by
the investigator of the very sensitive data that the "bad guy" was
attempting to acquire. Encryption may also be useful to protect
evidence written to removable storage media from tampering after it has
been written to disk. You didn't really look at that possibility.
However, if the "bad guy" has an advance copy of your tools and the time
to reverse engineer them, you will be toast in any event.
So here are my conclusions after reading your presentation:
1. Computer forensic investigators should acquire volatile evidence in a
forensically sound manner.
2. Memory acquisition tools should employ robust error logging and
cryptographic checksums to ensure the integrity of volatile evidence as
it is being transmitted and archived for future use.
3. Cryptographic checksums should be generated for volatile evidence as
early as possible in the acquisition process, preferrably in kernel mode.
4. Encryption should be employed during the transmission and archival of
digital evidence to preserve the confidentiality of sensitive data and
to reduce the attack surface.
My apologies to the Volatility mailing list for such a lengthy post.
However, I think that this topic is of sufficient importance to warrant
it.
I look forward to the publication of your PoC code bits.
Regards,
George M. Garner Jr.
President
GMG Systems, Inc.
10:15 a.m.
win32dd/win64dd has an option (/d) to generate Microsoft Crash Dumps
without using the crashdump! or KeBugCheck() functions.
On Mon, Jan 7, 2013 at 3:10 PM, George M. Garner Jr. <
ggarner_online(a)gmgsystemsinc.com> wrote:
Luka,
Good to hear from you and thanks again for drawing attention to the
importance of reliability issues in volatile evidence acquisition. The
robustness and reliability of memory acquisition tools has received scant
attention up until now, Yet, the acquisition phase is the most important
step. You can run an analysis tool and, if it fails, you can still run a
different analysis tool or patch the first analysis tool and run it again.
If an acquisition tool fails silently you are typically out of luck! Note
that the reliability issues addressed in your presentation are not unique
to memory acquisition but also apply to the live acquisition of other
storage media as well. Darren Bilby's 2006 Blackhat presentation (DDefy)
addressed both "live" memory and other storage acquisition. While your
current presentation focuses on physical memory acquisition, your
techniques could easily be applied to the "live" acquisition of fixed
storage media as well. And perhaps they should be so applied.
My apologies if my previous post seemed dismissive. The appropriate
response to a critical work is to view it critically, however. Darren
unfortunately abandoned his work on DDefy. My intention is not that you
abandon Dementia but that you improve it! I wouldn't worry too much about
giving the "bad guys" ideas, as they already are implementing plenty of
ideas, including some that haven't been mentioned yet.
Paper documents can be forged. Electronic documents can be tampered with.
Volatile evidence can be modified. We know that evidence can be
falsified. The question is, "What should we do about it?" That is why the
last slide in your presentation is the most important. Because it attempts
to provide us with a road map for the future, a way forward. Unfortunately
that is where I am having my greatest difficulty, as I will explain.
1. Acquisition tools should utilize drivers correctly, (i.e. they should
acquire memory into a kernel mode buffer and write that buffer to a file
from kernel mode)!
It is true that a tool such as win32dd will not be susceptible to user
mode DeviceIoControl/**NtDeviceIoControlFile or WriteFile/NtWriteFile
hooks. However, win32dd will be/is vulnerable to your kernel mode
NtWriteFile hook and to a file system filter driver (assuming that you are
implementing those methods correctly). Also, if you are able to install an
inline hook in the kernel mode NtWriteFile function, you ought equally to
be able to install inline hooks in NtMapViewOfSection, MmMapIoSpace and
MmMapMemoryDumpMdl. Or the same could be accomplished using IAT hooks,
like the driver verifier does it. Indeed, hooking those three functions
would cover all of the memory acquisition tools which you tested and render
output compression or encryption irrelevant. So how is the KM/KM paradigm
(e.g. win32dd) superior to other methods? Your PoC itself subverts both UM
and KM acqusition, assuming that you have implemented your NtWriteFile hook
correctly. If not then shame on you! Note that I am not saying this to
disparage win32dd. I am just saying that the approach taken by this tool
is not inherently more reliable than other methods.
2. Use hardware acquisition tools.
You're absolutely right regarding the 4 GB
limit. But from
an attacker's perspective, this method cannot be used for hiding
arbitrary object, and it might be difficult to "relocate" the
driver, allocations and all resources above the specified limit. <
Actually it is quite easy to load a driver or arbitrary object (e.g.
encryption key or key schedules) into a reserved physical address space
above 4 GiB. We are currently shipping some software to our customers that
is able to do precisely that. It wasn't written with the intention of
"cheating" firewire memory dumps; however, it should serve that purpose
equally well.
3. Use crash dumps (native!) instead of raw dumps.
I may have been mistaken about Sinowal removing itself from crashdumps
during the dump process. Of the publicly available rootkits, Rustock.C
appears to be the one most often cited as an example of this.
http://go.eset.com/us/**resources/white-papers/Yet_**
Another_Rustock_Analysis.pdf<http://go.eset.com/us/resources/white-paper…;
http://www.eicar.org/files/**lipovsky_eicar2010.pdf<http://www.eicar.org….
A very cursory static analysis of crashdmp!WriteBufferToDisk indicates
that BugCheckDumpIoCallback's are called before data is written to the dump
file.
uf crashdmp!WriteBufferToDisk
[...]
fffff880`00ac450a ff15f85b0000 call qword ptr
[crashdmp!_imp_**MmBuildMdlForNonPagedPool
(fffff880`00aca108)]
fffff880`00ac4510 448b4c2428 mov r9d,dword ptr [rsp+28h]
fffff880`00ac4515 448b442440 mov r8d,dword ptr [rsp+40h]
fffff880`00ac451a 488b542470 mov rdx,qword ptr [rsp+70h]
fffff880`00ac451f b801000000 mov eax,1
fffff880`00ac4524 488bce mov rcx,rsi
fffff880`00ac4527 660944245a or word ptr [rsp+5Ah],ax
fffff880`00ac452c e87f0e0000 call crashdmp!InvokeDumpCallbacks
(fffff880`00ac53b0)
and then:
uf crashdmp!InvokeDumpCallbacks
[...]
fffff880`00ac5442 488364242800 and qword ptr [rsp+28h],0
fffff880`00ac5448 33c0 xor eax,eax
fffff880`00ac544a 4889442430 mov qword ptr [rsp+30h],rax
fffff880`00ac544f 4889442438 mov qword ptr [rsp+38h],rax
fffff880`00ac5454 48834c2428ff or qword ptr
[rsp+28h],0FFFFFFFFFFFFFFFFh
fffff880`00ac545a 4c897c2430 mov qword ptr [rsp+30h],r15
fffff880`00ac545f 4489742438 mov dword ptr [rsp+38h],r14d
fffff880`00ac5464 8974243c mov dword ptr [rsp+3Ch],esi
fffff880`00ac5468 448d4818 lea r9d,[rax+18h] r9d =
sizeof(KBUGCHECK_DUMP_IO)
fffff880`00ac546c 4c8d442428 lea r8,[rsp+28h] ; r8 =
KBugcheckDumpIoBlock
fffff880`00ac5471 488bd3 mov rdx,rbx ; rdx = CallbackRecord
fffff880`00ac5474 8d4803 lea ecx,[rax+3]; ecx =
KbCallbackDumpIo
fffff880`00ac5477 ff5310 call qword ptr [rbx+10h]
The dump IO block contains a pointer to the data that is about to be
written to the dump file. Even if it turns out that this callback is being
called after data has been written to the dump file you could use the first
call (ReasonSpecificData->Type == KbDumpIoHeader) to remove the rootkit
from memory and then ignore subsequent calls.
The bigest problem with crash dumps is that the dump process discards so
much valuable evidence. Basically, you only get what is in the OS's
physical memory map with a crashdump. That will miss the real mode
interrupt vector table, many artifacts of ACPI rootkits and BIOS kits, and
rootkits loaded into unmanaged memory. In addition, crashdumps lack some
of the features that we expect from a trusted forensic tool such as
cryptographic checksums and robust error logging. Finally, the crashdump
overwrites the pagefile which itself may contain valuable evidence.
Once again, the superiority of crashdumps over other acquisition methods
appears to be largely due to a design decision on your part not to attack
crashdumps as vigorously as you might.
So where does that leave us? All hope is lost?
Paper documents can be forged. Electronic documents can be tampered with.
Volatile evidence can be modified. Computer forensics scarcely could go
on if the mere possibility or even fact of evidence tampering were
sufficient to invalidate the whole process. If evidence were inherently
reliable we would not need experts to acquire it or to apply a forensic
method to sift the facts. One of the major limitations of your
presentation is that it does not address the methods that professional
forensic tools use (or should use) to ensure the integrity of evidence as
it is being transmitted and archived for future use, e.g. robust logging
and cryptographic checksums. To be successful the "bad guy" must not only
alter volatile evidence, he must also keep the investigator from knowing
that the evidence has been altered. "Bad guys" try to "hide" or
alter
evidence in all sorts of different ways. But as Jesse Kornblum has pointed
out, the act of "hiding" itself produces artifacts which enhance the
visibility of the malware that is being hid (this is the "rootkit
paradox"). Before we begin to ask about malicious modification of volatile
evidence an even more basic question needs to be asked: Does the memory
acquisition tool implement a sound forensic method in the first place? If
the tool does not offer any means to ensure evidentiary integrity (e.g.
robust logging and cryptographic checksums) there is no need to ask about
malicious modification of evidence.
Once we are satisfied that a tool implements a sound forensic method we
would then like to see if the evidence adulteration is reflected in the
output of the tool in the form of log entries and/or modified cryptographic
checksum. In other words, I would like to see you acquire memory evidence
the way that a professional computer forensic "expert" would acquire
evidence and then see whether the PoC is able to subvert the acquisition
process without any indication in the form of log entries or altered
cryptographic checksums.
All of the techniques which you document, except for the user mode
DeviceIoControl hook, would be defeated simply by adopting a forensically
sound method. If any thing I draw the conclusion that a crytographic hash
of the data should be generated very early in the acquisition process, and
as close to the actual reading of the data as possible. If you were to
conclude that memory acquisition tools should generate cryptographic hashes
of the evidence in KM I would buy that! Cryptographic checksums might still
be defeated by NtMapViewOfSection, MmMapIoSpace and MmMapMemoryDumpMdl
hooks. But we can detect their operation using other means.
While not strictly a part of the acquisition process, robust analysis is
also a means to ensure evidentiary reliability. Even when robust logging
and cryptographic hashes fail, robust analysis still may be able to detect
anomalies which indicate evidence tampering. To draw a couple of analogies
from the world of disk forensics, "bad guys" often tamper with file system
MAC times. But then forensic investigators look for artifacts of altered
MAC times (or the tools which generated them) elsewhere in the file system
image. "Bad guys" also use file or disk wiping software (e.g. CCleaner) to
destroy digital evidence. Forensic experts then look for artifacts that
CCleaner was used on the system within the time frame that evidence is
believed to have been destroyed. How is using Dimentia to wipe certain
artifacts from memory different from using CCleaner to destroy disk
evidence, if I can find artifacts which show that Dimentia was used?
Inline hooks in a user mode process leave some definite artifacts even
after they have been removed. Kernel mode inline hooks (e.g. in
NtWriteFile) also are highly visible, and they don't work on 64-bit Windows
as you acknowledge. Even the lower disk filters used by most bootkits are
visible now that everyone knows to look for them.
In our own research we have not had much trouble coming up with ways to
"cheat" physical memory acquisition. What is difficult is finding ways to
"cheat" the acquisition process without anyone knowing about it (assuming
that a sound forensic method was applied during acquisition). To be
successful anti-forensic software must accomplish its purpose while leaving
very few traces behind. One approach may be to place the rootkit in a
memory location that the forensic tool does not acquire. For example, some
tools aren't able to acquire non-cached or write-combined memory on Windows
systems prior to Windows 7 (e.g. MDD). You might be able to "hide" a
rootkit from one of these tools simply by placing the rootkit in non-cached
or write-combined memory without any hooks whatsoever. (Well, actually,
only the appropriate entry in the PFN database needs to say that the memory
is non-cached or write-combined, and not the PTE entry itself.)
Robust logging and proper use of cryptographic checksums are an essential
attribute of a trusted forensic tool. If your PoC were to force forensic
tool vendors to implement sound forensic methods, or failing that, to show
the perils of not implementing those methods, it would do a great service
to the forensic community. To be fair you should not ignore suitable and
obvious methods of subverting KM acquisition methods. (Why is it that you
chose not to attack NtMapViewOfSection, MmMapIoSpace and MmMapMemoryDumpMdl
directly?) A very interesting extension of your work would be to modify
open source router firmware to strip malicious artifacts from the data
stream as it is being transmitted over the wire. Then see how many tools
notice that the evidence was modified. :-)
Just for the record, we strongly urge our customers to acquire evidence to
the network when the presence of malware is suspected. Malware that
spreads via removable storage devices is quite common now-a-days. Evidence
acquired to a local storage device needs special handling to avoid
spreading the infection to other computers on site and to avoid the
investigator becoming part of someone's botnet. We also strongly recommend
the use of content encryption (and not just channel encryption) when
acquiring over the network to prevent the disclosure by the investigator of
the very sensitive data that the "bad guy" was attempting to acquire.
Encryption may also be useful to protect evidence written to removable
storage media from tampering after it has been written to disk. You didn't
really look at that possibility. However, if the "bad guy" has an advance
copy of your tools and the time to reverse engineer them, you will be toast
in any event.
So here are my conclusions after reading your presentation:
1. Computer forensic investigators should acquire volatile evidence in a
forensically sound manner.
2. Memory acquisition tools should employ robust error logging and
cryptographic checksums to ensure the integrity of volatile evidence as it
is being transmitted and archived for future use.
3. Cryptographic checksums should be generated for volatile evidence as
early as possible in the acquisition process, preferrably in kernel mode.
4. Encryption should be employed during the transmission and archival of
digital evidence to preserve the confidentiality of sensitive data and to
reduce the attack surface.
My apologies to the Volatility mailing list for such a lengthy post.
However, I think that this topic is of sufficient importance to warrant it.
I look forward to the publication of your PoC code bits.
Regards,
George M. Garner Jr.
President
GMG Systems, Inc.
______________________________**_________________
Vol-users mailing list
Vol-users(a)volatilityfoundation.org
http://lists.volatilesystems.**com/mailman/listinfo/vol-users<http://lis…
10:20 a.m.
Bonjour Matthieu!
On 1/7/2013 10:15 AM, Matthieu Suiche wrote:> win32dd/win64dd has an
option (/d) to generate Microsoft Crash Dumps
without using the crashdump! or KeBugCheck()
functions.
Yes, but how does the change in format alter the function of Luka's
NtWriteFile hook, except to give him less information to scrub? Of
course you could roll your own IRP_MJ_WRITE and bypass NtWriteFile. But
then Luka could use his file system filter driver, or ask Peter
Kleissner for a copy of the open source Stoned bootkit and adapt his
lower disk/ACPI/ATAPI filter driver for the task.
The format of the output isn't the problem.
Regards,
George.
10:29 a.m.
I thought Luka was pointing out the Microsoft crash dump format because
it's easier to analyze with WinDbg etc. Not for acquisition reason, if it's
for acquisition reason that's exactly the same. That's why one of the
reason Microsoft is encouraging people to use 64-bits, even though of
course PatchGuard can still be bypassed. No security mechanism is perfect.
That's the cat and mouse game we all know very well. :)
On Mon, Jan 7, 2013 at 3:20 PM, George M. Garner Jr. <
ggarner_online(a)gmgsystemsinc.com> wrote:
Bonjour Matthieu!
On 1/7/2013 10:15 AM, Matthieu Suiche wrote:> win32dd/win64dd has an
option (/d) to generate Microsoft Crash Dumps
without using the crashdump! or KeBugCheck()
functions.
Yes, but how does the change in format alter the function of Luka's
NtWriteFile hook, except to give him less information to scrub? Of course
you could roll your own IRP_MJ_WRITE and bypass NtWriteFile. But then Luka
could use his file system filter driver, or ask Peter Kleissner for a copy
of the open source Stoned bootkit and adapt his lower disk/ACPI/ATAPI
filter driver for the task.
The format of the output isn't the problem.
Regards,
George.
______________________________**_________________
Vol-users mailing list
Vol-users(a)volatilityfoundation.org
http://lists.volatilesystems.**com/mailman/listinfo/vol-users<http://lis…
2:52 p.m.
Hi George,
thanks for your answer, it definitely shows that you're deeply in the
field and that I still have a lot to learn about:)
Here are my comments:
On Mon, Jan 7, 2013 at 4:10 PM, George M. Garner Jr.
<ggarner_online(a)gmgsystemsinc.com> wrote:
Luka,
Good to hear from you and thanks again for drawing attention to the
importance of reliability issues in volatile evidence acquisition. The
robustness and reliability of memory acquisition tools has received scant
attention up until now, Yet, the acquisition phase is the most important
step. You can run an analysis tool and, if it fails, you can still run a
different analysis tool or patch the first analysis tool and run it again.
If an acquisition tool fails silently you are typically out of luck! Note
that the reliability issues addressed in your presentation are not unique to
memory acquisition but also apply to the live acquisition of other storage
media as well. Darren Bilby's 2006 Blackhat presentation (DDefy) addressed
both "live" memory and other storage acquisition. While your current
presentation focuses on physical memory acquisition, your techniques could
easily be applied to the "live" acquisition of fixed storage media as well.
And perhaps they should be so applied.
I agree completely - this was just an extension of DDefy to memory
forensics (although it did posess some mem-related hiding abilities)
and addition of new features.
My apologies if my previous post seemed dismissive. The appropriate
response to a critical work is to view it critically, however. Darren
unfortunately abandoned his work on DDefy. My intention is not that you
abandon Dementia but that you improve it! I wouldn't worry too much about
giving the "bad guys" ideas, as they already are implementing plenty of
ideas, including some that haven't been mentioned yet.
Thanks:)
1. Acquisition tools should utilize drivers correctly, (i.e. they should
acquire memory into a kernel mode buffer and write that buffer to a file
from kernel mode)!
It is true that a tool such as win32dd will not be susceptible to user mode
DeviceIoControl/NtDeviceIoControlFile or WriteFile/NtWriteFile hooks.
However, win32dd will be/is vulnerable to your kernel mode NtWriteFile hook
and to a file system filter driver (assuming that you are implementing those
methods correctly). Also, if you are able to install an inline hook in the
kernel mode NtWriteFile function, you ought equally to be able to install
inline hooks in NtMapViewOfSection, MmMapIoSpace and MmMapMemoryDumpMdl. Or
the same could be accomplished using IAT hooks, like the driver verifier
does it. Indeed, hooking those three functions would cover all of the
memory acquisition tools which you tested and render output compression or
encryption irrelevant. So how is the KM/KM paradigm (e.g. win32dd) superior
to other methods? Your PoC itself subverts both UM and KM acqusition,
assuming that you have implemented your NtWriteFile hook correctly. If not
then shame on you! Note that I am not saying this to disparage win32dd. I
am just saying that the approach taken by this tool is not inherently more
reliable than other methods.
Dementia has two drivers - one is using NtWriteFile hook, the other
one is a FS minifilter. They are not used in parallel of course:)
Using any of these two drivers, Dementia is able to hide mentioned
artifacts from all acquisition tools I tested (Win32dd included),
except Winen and OSForensics.
Winen is creating EWF and it's tricky to modify the file as it's being
written (but not impossible). However, it transfers the memory pages
to user-mode in *clean* format, so I could use the user mode injection
(DeviceIoControl hook). Injection to Winen process is not implemented
at the moment, but it's trivial to extend Dementia in order to support
it.
OSForensics is doing something strange with the buffers being written
to the file. I had no time (and no motivation either) to check why
can't I access the buffer being written to the file (AV fault), but
it's again susceptible to UM injection.
KM/KM paradigm used ONLY by win32dd is better in many ways:
1. Buffer never leaves the kernel space - attacker cannot attack it
from the user-mode
2. Buffer never leaves the kernel space - no performance overhead of
communicating with the driver, copying the memory back to user-space
(or not, depending on the method used, but then you have additional
probes of UM buffer etc.)
3. Logging and/or encryption methods done in KM-only are again much
safer than doing it in user-mode (you mention it later in your answer)
It does not solve the problem, as I have demonstrated, but I think it
is simply unnecessary and worse approach to transfer the buffer back
to user-mode and then write it to dump.
I'm not completely sure how would you do sound acquisition using
crypto/logging mechanisms if someone hooks deeply in the kernel? By
the time you get your buffer, it has already been tampered with. Some
kind of cross-viewing could be applied, but I'm not quite sure about
that...
As a side note - win32dd is the fastest tool I tested, and it's my
definitive favorite:)
2. Use hardware acquisition tools.
Maybe I'm missing something here, but how would you move *an
arbitrary* object above 4 GB.
By arbitrary, I mean really arbitrary object, for example an
allocation in user-mode, network connection (i.e. socket), file
object, VAD, anything...
Wow, I completely forgot about Rustock, this is my fault:(
Does anyone have a short analysis of this part of the Rustock.C?
I was able to find the sample itself, but I really don't have the time
to reverse engineer the rootkit in search for this tiny functionality
I'm interested in:(
I'm really interested in how Rustock does it.
Very nice find, and fail for me not reading the Matthieu's and other's
presentations:/
You're absolutely right regarding the 4 GB
limit. But from
an attacker's perspective, this method cannot be used for hiding
arbitrary object, and it might be difficult to "relocate" the
driver, allocations and all resources above the specified limit. <
Actually it is quite easy to load a driver or arbitrary object (e.g.
encryption key or key schedules) into a reserved physical address space
above 4 GiB. We are currently shipping some software to our customers that
is able to do precisely that. It wasn't written with the intention of
"cheating" firewire memory dumps; however, it should serve that purpose
equally well. Once again, the superiority of crashdumps over other
acquisition methods
appears to be largely due to a design decision on your part not to attack
crashdumps as vigorously as you might.
You may be right.
However, as Matthieu mentioned in his post, win32dd is not utilizing
normal Windows crash-dump path - but Dementia modifies the crash-dump
produced by the win32dd in the same way it modifies raw dump produced
by any other tool:(
And yes, I did thought that, from a forensics perspective, crash-dumps
are more sound than raw dumps, because they use different I/O path:(
Will probably investigate this part further.
Once we are satisfied that a tool implements a sound forensic method we
would then like to see if the evidence adulteration is reflected in the
output of the tool in the form of log entries and/or modified cryptographic
checksum. In other words, I would like to see you acquire memory evidence
the way that a professional computer forensic "expert" would acquire
evidence and then see whether the PoC is able to subvert the acquisition
process without any indication in the form of log entries or altered
cryptographic checksums.
I'm not a computer forensic "expert" and unfortunately I don't have
the ability to test expensive memory acquisition tools.
I had the opportunity to work with EnCase for a brief period of time,
but it seems that memory analysis capabilities of EnCase are
relatively weak (various EnScripts, I didn't test them throughly) and
acquisition is completely based on Winen.
None of the tools I tested had such cryptographic and advanced logging
capabilities.
I had no chance to work with KnTTools, so I cannot say anything about KnTTools.
How is using Dimentia to wipe certain
artifacts from memory different from using CCleaner to destroy disk
evidence, if I can find artifacts which show that Dimentia was used? Inline
hooks in a user mode process leave some definite artifacts even after they
have been removed. Kernel mode inline hooks (e.g. in NtWriteFile) also are
highly visible, and they don't work on 64-bit Windows as you acknowledge.
Even the lower disk filters used by most bootkits are visible now that
everyone knows to look for them.
I could see some analogies between CCleaner and Dementia, but I
wouldn't say that they are equal (in the "cleaning-stuff" analogy) .
Dementia is operating in a different realm than the data being
collected. Although that sounds a bit pretentious, it's actually a
simple premise - it's not modifying the live machine, it is modifying
the data inside the "dead dump", and it can do anything it wants
inside the dump.
Although inline hooks are highly visible, they are not visible inside
the dump if I deliberately remove them:) And it's completely trivial
to restore the original opcodes if you're the one that did the
hooking.
Minifilters are noisy and visible as well, but again, Dementia could
remove that as well.
That being said, Dementia DOES NOT currently hide hooks or
minifilters, but could do so in the future.
That's why one of my conclusions was to perform "live" anti-rootkit
analysis, or cross-viewing.
I don't think that's a valuable or "sound" conclusion, because, as you
mentioned, AV's have been fighting with this for a long time.
One approach may be to place the rootkit in a
memory location that the forensic tool does not acquire. For example, some
tools aren't able to acquire non-cached or write-combined memory on Windows
systems prior to Windows 7 (e.g. MDD). You might be able to "hide" a
rootkit from one of these tools simply by placing the rootkit in non-cached
or write-combined memory without any hooks whatsoever. (Well, actually,
only the appropriate entry in the PFN database needs to say that the memory
is non-cached or write-combined, and not the PTE entry itself.)
Very cool, I didn't know that:)
To be fair you should not ignore suitable and
obvious methods of subverting KM acquisition methods. (Why is it that you
chose not to attack NtMapViewOfSection, MmMapIoSpace and MmMapMemoryDumpMdl
directly?)
I got that question after the talk:)
Here are some of the reasons (you'll probably find them foul in one
way or another, but anyway...):
1. File-write-interception is a generic method that covers all tools -
they have to write the dump in the end (unless they send it over the
network, which is when Dementia is currently screwed:))
2. File-writes gives me context - I can detect (easily? with adequate
accurracy?) that memory acquisition app is being used and it's going
to write the dump. Although MmMapMemoryDumpMdl is called rarely,
Nt(Mm)MapViewOfSection is called A LOT and it might be difficult to
detect whether dump is being written or not.
3. One hook is better than multiple hooks:)
4. No hooking in x64.
And I can hook deep, I have another PoC that does MmMapMemoryDumpMdl
hooking, but it's not generic enough. And it's not usable on x64.
Dementia "evolution" would probably go in the direction of deep
hooking on x86, while keeping the minifilter on x64.
A very interesting extension of your work would be to
modify
open source router firmware to strip malicious artifacts from the data
stream as it is being transmitted over the wire. Then see how many tools
notice that the evidence was modified. :-)
Cool, but I certainly don't have enough knowledge for that:)
Encryption may also be useful
to protect evidence written to removable storage media from tampering after
it has been written to disk. You didn't really look at that possibility.
That was not in scope of my research.
Encryption post-festum does not provide any protection against tools
such as Dementia.
So here are my conclusions after reading your presentation:
1. Computer forensic investigators should acquire volatile evidence in a
forensically sound manner.
Agree, but what does "forensically sound manner" actually mean in this context?
Forgive me for perhaps not knowing the basics...
2. Memory acquisition tools should employ robust error logging and
cryptographic checksums to ensure the integrity of volatile evidence as it
is being transmitted and archived for future uses
3. Cryptographic checksums should be generated for
volatile evidence as
early as possible in the acquisition process, preferrably in kernel mode.
4. Encryption should be employed during the transmission and archival of
digital evidence to preserve the confidentiality of sensitive data and to
reduce the attack surface.
Agree, but again, your view of memory could be polluted before you
even have the chance to calculate the checksum.
Yes, "could", but Dementia does prove it in a simplistic way
(DeviceIoControl hook).
I'm not sure where I'm actually heading with Dementia:) It was a cool
research for me, I learned quite a bit about Windows internals and
maybe raised awareness and pointed out some weak points with memory
acquisition process (that you were all probably already aware of, but
maybe since there was no tool to directly exploit them, it wasn't
perceived as "dangerous" enough).
I don't know my further steps. I'm wondering how deep and "under the
radar" can I go, but as Matthieu says - "That's the cat and mouse game
we all know very well."
Thank you for your thoughts and comments.
And sorry for using another account:)
Cheers,
Luka
7:57 a.m.
Hi Luka,
First let me thank you for contributing this important work to the
field of memory forensics. It is not often discussed how vulnerable
we, as digital investigators, are to subversion and manipulation. I
personally believe that to be an effective forensic investigator, one
has to have a very good idea how their tools can be subverted and
defeated. For example, I have done a workshop at the open source
digital forensic conference demonstrating among other things how
Volatility can be easily defeated on a live system. For educational
purposes, I have added write support to the winpmem memory acquisition
tool which allows volatility itself to write to arbitrary physical
memory. Since Volatility is really good at finding the artifacts,
turns out its also really good at manipulating these artifacts :-).
https://volatility.googlecode.com/svn/branches/scudette/docs/references.html
http://bit.ly/OSDFC_Memory_Workshop
This allows demonstration of many standard rootkit hiding techniques
using python scripts interacting with the Volatility library (e.g. we
demonstrate dkom, changing process names/permissions/VAD permissions
etc).
I believe this kind of critical examination is essential to advance
the state of the art. None of this is especially surprising as an
attacker with kernel access can always win. However, I believe that if
the field develops to the point that in order to effectively hide, the
malware needs to manipulate so many things that hiding become very
difficult and time consuming, not to mention degrading to system
stability.
On 7 January 2013 20:52, Luka Milkovic <milkovic.luka(a)gmail.com> wrote:
KM/KM paradigm used ONLY by win32dd is better in many
ways:
1. Buffer never leaves the kernel space - attacker cannot attack it
from the user-mode
It is true that the image is written from kernel mode, but a kernel
mode rootkit can easily intercept the kernel functions just like in
usermode. Usermode hooking can still be used to intercept the IO
control which triggers acquisition (for example some acquisition modes
are easier to subvert and a user mode tool can still force these to be
used by hooking the io control command).
2. Buffer never leaves the kernel space - no
performance overhead of
communicating with the driver, copying the memory back to user-space
(or not, depending on the method used, but then you have additional
probes of UM buffer etc.)
Our measurements show that buffering between kernel and user space
does not have a very significant impact. In particular, the linux pmem
driver does support direct mmap implementation (bascially its an mmap
into physical memory), but in practice there is no measurable
performance difference. I believe that on windows it is also possible
to set up direct MDL mappings into userspace which simply avoid any
copying between kernel and user space. I never spent the time to
investigate this since I am not seeing significant performance gains
when comparing winpmem to win32dd.
3. Logging and/or encryption methods done in KM-only
are again much
safer than doing it in user-mode (you mention it later in your answer)
That depends on your frame of reference. Safer is only a relative
term. To an attacker which already owns kernel space it is the same
amount of work to hook a kernel module or a user space program.
It does not solve the problem, as I have demonstrated,
but I think it
is simply unnecessary and worse approach to transfer the buffer back
to user-mode and then write it to dump.
I disagree. Passing the buffer back to userspace makes user mode
processing much simpler and more reliable since you do not need to do
much processing in kernel space. For example, it is possible to run
Volatility directly on the live system without taking an image first,
for a quick triage of the system - this is faster and more efficient
than taking a full image if all we want is to search for specific
sigs.
In practice it is trivial to encrypt/obfuscate the data between kernel
and user space which makes manipulating these buffers from userspace
by Dementia more difficult without intimate knowledge of how the
specific acquisition tool works.
The real challenge for Dementia the way I see it is to be able to
identify the DFIR tool in order to interfere with it. You can not
simply go around string replacing arbitrary data in buffers copied
between userspace and kernel space without being sure its not going to
adversely affect system stability. So you need to be pretty sure the
buffers represent physical memory.
Identifying commercial tools is pretty trivial - for example, all
tools need to be signed now so their signatures are predictable and
can be black listed. Perhaps many commercial tools have constant
strings (many have copyright strings) which can be used to identify
when they are inserted into the kernel.
The advantage of an open source acquisition tool is that it can be
easily changed/tailoured in such a way that malicious tools can not
easily identify it.
I'm not completely sure how would you do sound
acquisition using
crypto/logging mechanisms if someone hooks deeply in the kernel? By
the time you get your buffer, it has already been tampered with. Some
kind of cross-viewing could be applied, but I'm not quite sure about
that...
Indeed the exact APIs that are used to obtain the memory image can
also be hooked. The trick is to use APIs which are not obviously used
for the purpose of memory imaging. For example if you call certain
APIs which are commonly used for other things it will be dangerous for
Dementia to automatically hook because it might interfere with another
legitimate driver (e.g. setting a memory mapped io region or
something).
You may be right.
However, as Matthieu mentioned in his post, win32dd is not utilizing
normal Windows crash-dump path - but Dementia modifies the crash-dump
produced by the win32dd in the same way it modifies raw dump produced
by any other tool:(
I believe win32dd is simply writing a crash dump compatible file (e.g.
the same file format) so it can be used with windbg (winpmem does that
too), but it still uses the same acquisition APIs to actually get the
data.
And yes, I did thought that, from a forensics
perspective, crash-dumps
are more sound than raw dumps, because they use different I/O path:(
Having multiple acquisition techniques helps because it makes it
harder for the attacker to cover off on all the different
APIs/methods, but ultimately you will never know if the specific
rootkit which is infecting this machine is hooking those APIs you are
about to use :-).
> Once we are satisfied that a tool implements a
sound forensic method we
> would then like to see if the evidence adulteration is reflected in the
> output of the tool in the form of log entries and/or modified cryptographic
> checksum. In other words, I would like to see you acquire memory evidence
> the way that a professional computer forensic "expert" would acquire
> evidence and then see whether the PoC is able to subvert the acquisition
> process without any indication in the form of log entries or altered
> cryptographic checksums.
There are a couple of goals in a forensic acquisition. The first is to
try to find evidence of compromise, and the second is to get
information about the compromise. While it is difficult to leave no
traces at all - it is trivial to break acquisition and analysis tools
in such a way that, although they can be pretty sure the system is
owned, it is far beyond the skills of the average responder to
actually get a copy of the malware. For example, by breaking the
system in such a way that any acquisition tool fails to work at all -
this can get the responder to be pretty sure the system is owned, but
they still can not acquire the image or evidence. Coupled with a
really good persistence mechanism (so its hard to locate the malware
on disk) and most responders will just rebuild the box and move on.
Clearly any assumption of the soundness of the tool really depends on
the assumption that the system is not compromised and operates
normally. Once you consider the possibility that the system is owned,
there is no point talking about forensic soundness since subversion is
pretty much assumed.
> How is using Dimentia to wipe certain
> artifacts from memory different from using CCleaner to destroy disk
> evidence, if I can find artifacts which show that Dimentia was used? Inline
> hooks in a user mode process leave some definite artifacts even after they
> have been removed. Kernel mode inline hooks (e.g. in NtWriteFile) also are
> highly visible, and they don't work on 64-bit Windows as you acknowledge.
> Even the lower disk filters used by most bootkits are visible now that
> everyone knows to look for them.
Indeed, this is why the field of memory forensics is rapidly evolving.
We must consider the possibility that our own tools and techniques are
used against us - those artifacts that you are finding using
Volatility (and other tools) can also be found by the bad guys and
cleaned up just as easily. It is an arms race :-).
Anyway, thanks Luka for an interesting talk and a nice discussion.
Michael.
9:44 a.m.
Michael,
On 1/8/2013 7:57 AM, Michael Cohen wrote:
The real challenge for Dementia the way I see it is to
be able to
identify the DFIR tool in order to interfere with it.
See the section entitled "Parsing Certificates" in Peter Kleissner,
"Creating an anti-AV scanner...and blocking AV's,"
http://www.stoned-vienna.com/downloads/Creating%20an%20Anti-AV%20scanner%20….
Regards,
gmg.
3:07 p.m.
Hi Michael,
first of all, thanks for your support:)
My measurements using the same buffer sizes (when applicable, I cannot
use arbitrary buffer length for all tools) show that the difference is
around 5% , which I agree is not noticable and definitely not a
significant performance gain.
What makes a bigger difference is the size of the buffer itself. Many
tools are using PAGE_SIZE as the buffer size, win32dd and winpmem are
one of the rare tools that are using big buffers by default (and I
think that's a good thing). Difference can then be even up to 100%.
I have not proved this assumption, because some other variables could
influence the speed of the acquisition and I didn't reverse engineer
every single driver to see which transfer methods and acquisition
methods are being used.
I agree, but as I said, to an attacker who "owns" the acquisition tool
in user space (via binary planting or some other methods), access to
kernel space might be much harder.
I agree with you completely regarding reliability and simplicity, but
I'm still not so convinced about the security. At least you make it
one bit harder for an attacker (who also has the same reliability and
simplicity issues in his driver:) ).
For example, I have done a workshop at the open
source
digital forensic conference demonstrating among other things how
Volatility can be easily defeated on a live system. For educational
purposes, I have added write support to the winpmem memory acquisition
tool which allows volatility itself to write to arbitrary physical
memory. Since Volatility is really good at finding the artifacts,
turns out its also really good at manipulating these artifacts :-).
https://volatility.googlecode.com/svn/branches/scudette/docs/references.html
http://bit.ly/OSDFC_Memory_Workshop
Very cool, I already saw that some time ago and it's a very nice intro
and walkthrough for Volatility.
On 7 January 2013 20:52, Luka Milkovic
<milkovic.luka(a)gmail.com> wrote:
The only thing that currently really puts both paradigms in more or
less equal positions from an attacker's perspective is that you need
admin privileges for UM injection/hooking, because the acquisition app
itself needs admin privs.
But there are other techniques available in UM that could subvert the
correctness of the process and attack acquisition applications from a
less privileged account.
For example, I made some quick test for binary planting
vulnerabilities and almost all acquisition tools that I tested are
vulnerable to binary planting. Winpmem is actually the only tested
tool which is not vulnerable to binary planting, while Memoryze and
OSForensics have large number of such vulnerabilities.
I don't consider BP such a big issue here (although it is not
something that should be completely ignored), but it further supports
"KM is better" theory I'm advocating.
Of course KM is not a complete solution, but the attack surface is
somewhat lowered.
As a side note - Volatility on Windows is also searching for DLLs and
Python code in the current directory first. I wonder if someone made a
pyc planting:)
KM/KM paradigm used ONLY by win32dd is better in
many ways:
1. Buffer never leaves the kernel space - attacker cannot attack it
from the user-mode
It is true that the image is written from kernel mode, but a kernel
mode rootkit can easily intercept the kernel functions just like in
usermode. Usermode hooking can still be used to intercept the IO
control which triggers acquisition (for example some acquisition modes
are easier to subvert and a user mode tool can still force these to be
used by hooking the io control command). 2. Buffer never leaves the kernel space - no
performance overhead of
communicating with the driver, copying the memory back to user-space
(or not, depending on the method used, but then you have additional
probes of UM buffer etc.)
Our measurements show that buffering between kernel and user space
does not have a very significant impact. In particular, the linux pmem
driver does support direct mmap implementation (bascially its an mmap
into physical memory), but in practice there is no measurable
performance difference. I believe that on windows it is also possible
to set up direct MDL mappings into userspace which simply avoid any
copying between kernel and user space. I never spent the time to
investigate this since I am not seeing significant performance gains
when comparing winpmem to win32dd. 3. Logging and/or encryption methods done in
KM-only are again much
safer than doing it in user-mode (you mention it later in your answer)
That depends on your frame of reference. Safer is only a relative
term. To an attacker which already owns kernel space it is the same
amount of work to hook a kernel module or a user space program. It does not solve the problem, as I have
demonstrated, but I think it
is simply unnecessary and worse approach to transfer the buffer back
to user-mode and then write it to dump.
I disagree. Passing the buffer back to userspace makes user mode
processing much simpler and more reliable since you do not need to do
much processing in kernel space. For example, it is possible to run
Volatility directly on the live system without taking an image first,
for a quick triage of the system - this is faster and more efficient
than taking a full image if all we want is to search for specific
sigs.
In practice it is trivial to encrypt/obfuscate the data between kernel
and user space which makes manipulating these buffers from userspace
by Dementia more difficult without intimate knowledge of how the
specific acquisition tool works.
I agree.
The real challenge for Dementia the way I see it is to be able to
identify the DFIR tool in order to interfere with it.
Exactly!
It worries me that many infosec specialists (not necessarily forensic
experts) use acquisition tools without even minimal precautions - they
download the tool, put it on the USB stick and run the tool as-is,
without even renaming it. Detecting the acquisition application by its
name is one of the first filters used by Dementia and it is as dumb as
it gets:)
There are plenty of ways to detect DFIR tool, you mentioned some of
them and George gave a link on Peter Kleissner's whitepaper. I'm
thinking about implementing some of them, but don't have the time at
the moment...
It's heuristics in the end, and you're dead right that some commercial
applications are much less flexible here.
For example, by breaking the
system in such a way that any acquisition tool fails to work at all -
this can get the responder to be pretty sure the system is owned, but
they still can not acquire the image or evidence. Coupled with a
really good persistence mechanism (so its hard to locate the malware
on disk) and most responders will just rebuild the box and move on.
:) When I was thinking about writing tool like Dementia, a friend of
mine told me it's completely useless to create such tool:) His opinion
is that a tool which detects an installation of DFIR app driver or
launch of DFIR application and immediately crashes the machine or
reboots it (via HalReturnToFirmware or smth) is even more effective
then Dementia-like tools:)
This should not be suspicious to analyst, because he would think that
the application itself is causing the reboot.
We must consider the possibility that our own tools and techniques are
used against us - those artifacts that you are finding using
Volatility (and other tools) can also be found by the bad guys and
cleaned up just as easily. It is an arms race :-).
Actually, I was thinking about utilizing Volatility to work for me in
Dementia's hiding job - Volatility would detect the artifacts and I
would just need to remove them from the dump.
I abandoned this idea since I would need to sanitize the buffers on
the fly and would need to hack Volatility significantly. I'm not a
good Pyhton developer, so that would take some time:)
Anyway, thanks Luka for an interesting talk and a nice discussion.
Michael.
Thanks to you also Michael, it was an interesting discussion:)
Cheers,
Luka
12:29 p.m.
Luka,
On 1/7/2013 2:52 PM, Luka Milkovic wrote:
None of the tools I tested had such cryptographic and
advanced logging
capabilities.
Moonsols Community Edition appears to support a "/s" option which will
generate a cryptographic checksum which may be compared to the actual
checksum of the output file. While it is true, as you say, that
cryptographic checksums will not detect tampering which occurred prior
to the checksum's creation, adding the "/s" option should detect all of
the kernel mode attacks which you document. It will force you to attack
along the read path (NtMapViewOfSection, MmMapIoSpace and
MmMapMemoryDumpMdl) or else to fake both the content and the checksum.
I am assuming that Matthieu is generating the checksum before writing
the data to disk and not from the output file after it has been written.
+1 for Matthieu (maybe).
Once you consider the possibility that the system is
owned,
there is no point talking about forensic soundness since subversion
is pretty much assumed. <
If a "forensic" tool is unable reliably to acquire volatile evidence
from an uncompromised system there is no reason to believe that it will
perform any better in the face of sophisticated malware. The API's
above may fail as part of the ordinary operation of the operating
system. If an acquisition tool is unaware that the API has failed or is
internally aware but fails silently the investigator will be left to
guess whether the contents of the evidence file are an accurate
reflection of the state of physical memory at a given moment in time or
a reflection of the failure on the part of the acquisition process.
Failure to implement a sound forensic process is exploitable, moreover,
because a malicious entity has only to mimic a normal read error in the
right location to "hide" from the hapless investigator. Just because
there may be some sophisticated malware out there will defeat even the
most well designed acquisition tool is no excuse for allowing yourself
to become a victim of every script kiddie that comes along!
Regards,
George.
2:49 p.m.
Hi George,
On Wed, Jan 9, 2013 at 6:29 PM, George M. Garner Jr.
<ggarner_online(a)gmgsystemsinc.com> wrote:
Moonsols Community Edition appears to support a "/s" option which will
generate a cryptographic checksum which may be compared to the actual
checksum of the output file. While it is true, as you say, that
cryptographic checksums will not detect tampering which occurred prior to
the checksum's creation, adding the "/s" option should detect all of the
kernel mode attacks which you document. It will force you to attack along
the read path (NtMapViewOfSection, MmMapIoSpace and MmMapMemoryDumpMdl) or
else to fake both the content and the checksum. I am assuming that Matthieu
is generating the checksum before writing the data to disk and not from the
output file after it has been written. +1 for Matthieu (maybe).
Unfortunately, that's not the case:(
The reason is that the checksum is calculated on the buffer *after* it
has been written to the file (basically the next call after
ZwWriteFile).
When Dementia is used, checksum of the dump created by win32dd equals
to the one printed out in the console.
Matthieu, what do you think about making the checksum before writing
the buffer to dump?
Cheers,
Luka
3:04 p.m.
Unfortunately, that's not the case:(
The reason is that the checksum is calculated on the buffer *after* it
has been written to the file (basically the next call after
ZwWriteFile).
When Dementia is used, checksum of the dump created by win32dd equals
to the one printed out in the console.
Zut alors! This does point out the value of tool testing and not simply
relying upon manufacturer's representations of what their tools do.
Dimentia has already served a very useful purpose for the forensic
community. Keep up the good work!
3:06 p.m.
I just read the email quickly.
The checksum is being computed before writting on disk. There is no reason
to read the file to compute the checksum.
buffer = readmem(pa, size)
chksum += checksum(buffer, size)
write(buffer, size)
On Wed, Jan 9, 2013 at 11:49 AM, Luka Milkovic <milkovic.luka(a)gmail.com>wrote:
Hi George,
On Wed, Jan 9, 2013 at 6:29 PM, George M. Garner Jr.
<ggarner_online(a)gmgsystemsinc.com> wrote:
Moonsols Community Edition appears to support a "/s" option which will
generate a cryptographic checksum which may be compared to the actual
checksum of the output file. While it is true, as you say, that
cryptographic checksums will not detect tampering which occurred prior to
the checksum's creation, adding the "/s" option should detect all of the
kernel mode attacks which you document. It will force you to attack
along
the read path (NtMapViewOfSection, MmMapIoSpace
and MmMapMemoryDumpMdl)
or
else to fake both the content and the checksum. I
am assuming that
Matthieu
is generating the checksum before writing the
data to disk and not from
the
output file after it has been written. +1 for
Matthieu (maybe).
Unfortunately, that's not the case:(
The reason is that the checksum is calculated on the buffer *after* it
has been written to the file (basically the next call after
ZwWriteFile).
When Dementia is used, checksum of the dump created by win32dd equals
to the one printed out in the console.
Matthieu, what do you think about making the checksum before writing
the buffer to dump?
Cheers,
Luka
_______________________________________________
Vol-users mailing list
Vol-users(a)volatilityfoundation.org
http://lists.volatilityfoundation.org/mailman/listinfo/vol-users
3:23 p.m.
Matthieu,
hmm, this is a bit strange.
Here is an output from win32dd (community edition, version
1.3.1.20100417, downloaded from your website some minutes ago):
-- win32dd --
Acquisition finished at: [2013-01-09 (YYYY-MM-DD) 20:07:49 (UTC)]
Time elapsed: 0:11 minutes:seconds (11 secs)
Created file size: 205520896 bytes ( 196 Mb)
NtStatus (troubleshooting): 0x00000000
Total of written pages: 50061
Total of inacessible pages: 0
Total of accessible pages: 50061
----SHA-256: 81454EFC92D357C68CE778FC3D74C22C3C4192D6A3B6F048102D97A95DAD0BCE----
-- end_of_win32dd --
Checksum of dump returns: SHA256:
81454efc92d357c68ce778fc3d74c22c3c4192d6a3b6f048102d97a95dad0bce
So the checksums are the same.
When reverse engineering the driver (sorry), I see this:
...
.text:00011009 push eax ; ByteOffset
.text:0001100A push esi ; Length
.text:0001100B push [ebp+Mdl] ; Buffer
.text:00011011 lea eax, [ebp+IoStatusBlock]
.text:00011017 push eax ; IoStatusBlock
.text:00011018 push ebx ; ApcContext
.text:00011019 push ebx ; ApcRoutine
.text:0001101A push ebx ; Event
.text:0001101B push [ebp+FileHandle] ; FileHandle
.text:00011021 call ds:ZwWriteFile ;
ZwWriteFile(Handle, NULL, NULL, NULL, ioStatusBlock, Buffer, Length,
byteOffset, NULL) - write the file
.text:00011027 mov [ebp+retStatus], eax
.text:0001102D cmp eax, 103h
.text:00011032 jnz short loc_1106C
.text:00011034 push ebx ; Timeout
.text:00011035 push ebx ; Alertable
.text:00011036 push [ebp+FileHandle] ; Handle
.text:0001103C call ds:NtWaitForSingleObject
.text:00011042 jmp short loc_11066
.text:00011044 ;
---------------------------------------------------------------------------
.text:00011044
.text:00011044 loc_11044: ; CODE XREF:
sub_10D38+2C8j
.text:00011044 cmp [ebp+dwPort?], 1
.text:0001104B jnz short loc_1106C
.text:0001104D lea eax, [ebp+var_1FC]
.text:00011053 push eax ; int
.text:00011054 push esi ; Length
.text:00011055 push [ebp+Mdl] ; Mdl
.text:0001105B push [ebp+FileObject] ; FileObject
.text:00011061 call sub_16D8E
.text:00011066
.text:00011066 loc_11066: ; CODE XREF:
sub_10D38+30Aj
.text:00011066 mov [ebp+retStatus], eax
.text:0001106C
.text:0001106C loc_1106C: ; CODE XREF:
sub_10D38+2FAj
.text:0001106C ; sub_10D38+313j
.text:0001106C cmp [ebp+retStatus], ebx
.text:00011072 jl loc_11297
.text:00011078 mov eax, [ebp+dwHashAlgorithm] ;
check whether hash algorithm is being used for checksumming
.text:0001107E dec eax
.text:0001107F jz short loc_110AE ; user specified SHA1
.text:00011081 dec eax
.text:00011082 jz short loc_1109C ; user specified MD5
.text:00011084 dec eax
.text:00011085 jnz short loc_110C1 ; no checksum
.text:00011087 push esi ; size_t
.text:00011088 push [ebp+Mdl] ; void *
.text:0001108E lea eax, [ebp+var_D4]
.text:00011094 push eax ; int
.text:00011095 call SHA256Sum ; do a SHA256
checksum over the buffer
.text:0001109A jmp short loc_110C1
.text:0001109C ;
---------------------------------------------------------------------------
.text:0001109C
.text:0001109C loc_1109C: ; CODE XREF:
sub_10D38+34Aj
.text:0001109C push esi
.text:0001109D push [ebp+Mdl]
.text:000110A3 lea eax, [ebp+var_6C]
.text:000110A6 push eax
.text:000110A7 call MD5Sum ; do a MD5Sum over the buffer
.text:000110AC jmp short loc_110C1
.text:000110AE ;
---------------------------------------------------------------------------
.text:000110AE
.text:000110AE loc_110AE: ; CODE XREF:
sub_10D38+347j
.text:000110AE push esi ; size_t
.text:000110AF push [ebp+Mdl] ; void *
.text:000110B5 lea eax, [ebp+var_1B0]
.text:000110BB push eax ; int
.text:000110BC call SHA1Sum ; do a SHA1Sum
Some of the annotations could be wrong because this was done in a hurry:)
But it seems that the checksums are calculated after the write. This
pattern repeats itself (depending on the method used for storing the
dump - whether the dump is stored locally or transferred over the
network).
So, the file is not read before comparing the checksum, but the
checksum is performed buffer-by-buffer before writing it to disk.
Maybe things are different in professional version, but I don't have
access to it:(
Cheers,
Luka
8:26 p.m.
New subject: Dementia in the news
Hi Luka! First, thank you, Luke, for all of your efforts!!! I very much appreciate
them!
Also, thank you to the others on this list that have contributed to this thread!! I have
enjoyed reading each one of them!
I do think this discussion is important and appropriate here!
I got this today. I send this for info to all.
http://www.dfinews.com/news/dementia-wipes-out-attacker-footprints-memory?e…
I have not read or evaluated this article yet, this is just to inform all here on this
list.
Best,
Mike Lambert
4326
days inactive
4331
days old
vol-users@lists.volatilityfoundation.org
14 comments
6 participants
participants (6)
-
George M. Garner Jr. -
Luka Milkovic -
Luka Milkovic -
Matthieu Suiche -
Michael Cohen -
Mike Lambert