Hi Luka,
First let me thank you for contributing this important work to the
field of memory forensics. It is not often discussed how vulnerable
we, as digital investigators, are to subversion and manipulation. I
personally believe that to be an effective forensic investigator, one
has to have a very good idea how their tools can be subverted and
defeated. For example, I have done a workshop at the open source
digital forensic conference demonstrating among other things how
Volatility can be easily defeated on a live system. For educational
purposes, I have added write support to the winpmem memory acquisition
tool which allows volatility itself to write to arbitrary physical
memory. Since Volatility is really good at finding the artifacts,
turns out its also really good at manipulating these artifacts :-).
https://volatility.googlecode.com/svn/branches/scudette/docs/references.html
http://bit.ly/OSDFC_Memory_Workshop
This allows demonstration of many standard rootkit hiding techniques
using python scripts interacting with the Volatility library (e.g. we
demonstrate dkom, changing process names/permissions/VAD permissions
etc).
I believe this kind of critical examination is essential to advance
the state of the art. None of this is especially surprising as an
attacker with kernel access can always win. However, I believe that if
the field develops to the point that in order to effectively hide, the
malware needs to manipulate so many things that hiding become very
difficult and time consuming, not to mention degrading to system
stability.
On 7 January 2013 20:52, Luka Milkovic <milkovic.luka(a)gmail.com> wrote:
KM/KM paradigm used ONLY by win32dd is better in many
ways:
1. Buffer never leaves the kernel space - attacker cannot attack it
from the user-mode
It is true that the image is written from kernel mode, but a kernel
mode rootkit can easily intercept the kernel functions just like in
usermode. Usermode hooking can still be used to intercept the IO
control which triggers acquisition (for example some acquisition modes
are easier to subvert and a user mode tool can still force these to be
used by hooking the io control command).
2. Buffer never leaves the kernel space - no
performance overhead of
communicating with the driver, copying the memory back to user-space
(or not, depending on the method used, but then you have additional
probes of UM buffer etc.)
Our measurements show that buffering between kernel and user space
does not have a very significant impact. In particular, the linux pmem
driver does support direct mmap implementation (bascially its an mmap
into physical memory), but in practice there is no measurable
performance difference. I believe that on windows it is also possible
to set up direct MDL mappings into userspace which simply avoid any
copying between kernel and user space. I never spent the time to
investigate this since I am not seeing significant performance gains
when comparing winpmem to win32dd.
3. Logging and/or encryption methods done in KM-only
are again much
safer than doing it in user-mode (you mention it later in your answer)
That depends on your frame of reference. Safer is only a relative
term. To an attacker which already owns kernel space it is the same
amount of work to hook a kernel module or a user space program.
It does not solve the problem, as I have demonstrated,
but I think it
is simply unnecessary and worse approach to transfer the buffer back
to user-mode and then write it to dump.
I disagree. Passing the buffer back to userspace makes user mode
processing much simpler and more reliable since you do not need to do
much processing in kernel space. For example, it is possible to run
Volatility directly on the live system without taking an image first,
for a quick triage of the system - this is faster and more efficient
than taking a full image if all we want is to search for specific
sigs.
In practice it is trivial to encrypt/obfuscate the data between kernel
and user space which makes manipulating these buffers from userspace
by Dementia more difficult without intimate knowledge of how the
specific acquisition tool works.
The real challenge for Dementia the way I see it is to be able to
identify the DFIR tool in order to interfere with it. You can not
simply go around string replacing arbitrary data in buffers copied
between userspace and kernel space without being sure its not going to
adversely affect system stability. So you need to be pretty sure the
buffers represent physical memory.
Identifying commercial tools is pretty trivial - for example, all
tools need to be signed now so their signatures are predictable and
can be black listed. Perhaps many commercial tools have constant
strings (many have copyright strings) which can be used to identify
when they are inserted into the kernel.
The advantage of an open source acquisition tool is that it can be
easily changed/tailoured in such a way that malicious tools can not
easily identify it.
I'm not completely sure how would you do sound
acquisition using
crypto/logging mechanisms if someone hooks deeply in the kernel? By
the time you get your buffer, it has already been tampered with. Some
kind of cross-viewing could be applied, but I'm not quite sure about
that...
Indeed the exact APIs that are used to obtain the memory image can
also be hooked. The trick is to use APIs which are not obviously used
for the purpose of memory imaging. For example if you call certain
APIs which are commonly used for other things it will be dangerous for
Dementia to automatically hook because it might interfere with another
legitimate driver (e.g. setting a memory mapped io region or
something).
You may be right.
However, as Matthieu mentioned in his post, win32dd is not utilizing
normal Windows crash-dump path - but Dementia modifies the crash-dump
produced by the win32dd in the same way it modifies raw dump produced
by any other tool:(
I believe win32dd is simply writing a crash dump compatible file (e.g.
the same file format) so it can be used with windbg (winpmem does that
too), but it still uses the same acquisition APIs to actually get the
data.
And yes, I did thought that, from a forensics
perspective, crash-dumps
are more sound than raw dumps, because they use different I/O path:(
Having multiple acquisition techniques helps because it makes it
harder for the attacker to cover off on all the different
APIs/methods, but ultimately you will never know if the specific
rootkit which is infecting this machine is hooking those APIs you are
about to use :-).
> Once we are satisfied that a tool implements a
sound forensic method we
> would then like to see if the evidence adulteration is reflected in the
> output of the tool in the form of log entries and/or modified cryptographic
> checksum. In other words, I would like to see you acquire memory evidence
> the way that a professional computer forensic "expert" would acquire
> evidence and then see whether the PoC is able to subvert the acquisition
> process without any indication in the form of log entries or altered
> cryptographic checksums.
There are a couple of goals in a forensic acquisition. The first is to
try to find evidence of compromise, and the second is to get
information about the compromise. While it is difficult to leave no
traces at all - it is trivial to break acquisition and analysis tools
in such a way that, although they can be pretty sure the system is
owned, it is far beyond the skills of the average responder to
actually get a copy of the malware. For example, by breaking the
system in such a way that any acquisition tool fails to work at all -
this can get the responder to be pretty sure the system is owned, but
they still can not acquire the image or evidence. Coupled with a
really good persistence mechanism (so its hard to locate the malware
on disk) and most responders will just rebuild the box and move on.
Clearly any assumption of the soundness of the tool really depends on
the assumption that the system is not compromised and operates
normally. Once you consider the possibility that the system is owned,
there is no point talking about forensic soundness since subversion is
pretty much assumed.
> How is using Dimentia to wipe certain
> artifacts from memory different from using CCleaner to destroy disk
> evidence, if I can find artifacts which show that Dimentia was used? Inline
> hooks in a user mode process leave some definite artifacts even after they
> have been removed. Kernel mode inline hooks (e.g. in NtWriteFile) also are
> highly visible, and they don't work on 64-bit Windows as you acknowledge.
> Even the lower disk filters used by most bootkits are visible now that
> everyone knows to look for them.
Indeed, this is why the field of memory forensics is rapidly evolving.
We must consider the possibility that our own tools and techniques are
used against us - those artifacts that you are finding using
Volatility (and other tools) can also be found by the bad guys and
cleaned up just as easily. It is an arms race :-).
Anyway, thanks Luka for an interesting talk and a nice discussion.
Michael.