Interesting message, especially the part explaining the differences between imaging and sampling. I also contacted the authors and express my feelings about the lack of accuracy of the document, and explained at the same time that the highlighted "flaws" aren't really new or unique and that it would have been been for the authors to also mentioned to non-developers who will read this paper thinking they are providing solutions that as long as you are using software acquisition based *regardless* of the acquisition methods you gonna use, if you are running as the same level of above any malicious program you are *always* running under a risk and there is *no* solutions to that. If an attacker has (physical) access to your box, it's game over. Best, On Mon, Aug 12, 2013 at 7:46 AM, George M. Garner Jr. &lt;ggarner_online(a)gmgsystemsinc.com <mailto:ggarner_online@gmgsystemsinc.com>> wrote: This post is not specifically about Volatility; however, I know that the topic will be of interest to many list members. Reliable memory analysis depends on reliable memory acquisition. No matter how good an analysis tool is, it can only analyze what is in the memory "dump." If the memory acquisition tool produces an incomplete or erroneous memory dump the analysis will also fail or be misleading. It is thus with pleasure that I note the progress that is being made toward developing a critical framework for evaluating and testing memory acquisition tools. The latest contribution is by Stefan Vömel and Johannes Stüttgen whose paper, "An evaluation platform for forensic memory acquisition software" was presented at the recent DFRWS conference and may be downloaded from the following link: http://www.dfrws.org/2013/proceedings/DFRWS2013-11.pdf. The authors adopt an approach which they call "white-box testing" whereby the authors modify the source code for various open source tools (win32dd, mdd, winpmem) to insert hypercalls at various locations in the acquisition process. The hypercalls inform the test platform of various important system events and operations and inspect the state of the subject system at the moment of the hypercall. The state as recorded by the hypercalls is then used as a metric to evaluate the reliability (i.e. "correctness") of the tool which is under test. The approach has a number of significant limitations which the authors acknowledge, the most significant of which is that they require the source code which must be modified in order to perform the test. Most commercial tool vendors do not provide the source code to their memory acquisition tools. Even one of the open source tools tested by Vömel and Stüttgen, win32dd, is an old version of the current closed-source Moonsols tool which contains many bug fixes which are not in the open source precursor. As far as I am aware the MDD tool is no longer supported or under active development. Michael Cohen's winpmem is the only currently supported tool that the authors were able to test. Another significant limitation is that the test platform is tied to a highly customized version of the Bochs x86 PC emulator. The test platform is restricted to 32-bit versions of Windows with not more than 2 GiB of memory. The acquisition of physical memory from systems equipped with more than 4 GiB of memory and from 64-bit versions of Microsoft Windows are areas where memory acquisition tool vendors have stumbled in the past. Possibly all contemporary memory acquisition tools handle 64-bit systems and systems with more than 4 GiB of memory correctly; however, we would like to be able to test this and not rely solely on faith. One limitation which the authors do not discuss is the impact of restricting the test platform to a particular VM (i.e. Bochs). In our experience VMs provide a much more highly predictable and stable environment than real hardware and may not be a good indication of how a memory acquisition tool will perform on real hardware. In addition, as was mentioned previously on this list, different VM manufacturers have chosen to emulate very different PC designs. How a tool performs on VMWare may not be a good indicator of how the same tool will perform on Microsoft Hyper-V or VirtualBox or KVM. Also, the authors do not acknowledge the possibility that memory acquisition tools may perform differently on different versions of the Microsoft operating system. Each new version of the Microsoft operating system has brought changes to the Windows memory manager, in some cases significant changes. Currently, we are (out of necessity) using acquisition methods that differ in varying degrees on Microsoft Windows XP, Windows 2003 SP1 and later, Windows Vista, Windows 7 and Windows 8. For Windows 8.1 Preview we have found it necessary to invent new methods that are different from all of the methods which we previously employed. Finally, the authors do not articulate the theoretical framework for forensic volatile memory acquisition which serves as the basis for their notion of "correctness." Historically, computer forensic experts have evaluated the acquisition of volatile memory as an "imaging" process. Most computer forensic experts were familiar with the imaging of "dead" hard drives. It was natural to assume memory acquisition was doing much the same thing. The problem is that a running computer system is by nature a stochastic process (more precisely, it is a "continuous time stochastic process") which cannot be "imaged." It can only be sampled. Sampling is a common technique in forensic science, other than in computer forensics. Computer forensics is somewhat unique among forensic "sciences" in its almost exclusive reliance upon "imaging" as a standard of reliability. It is difficult not to note the severe tension which exists between this theoretic framework and the reality of modern computer design. Not only is "live" imaging of hard drives now commonplace for practical reasons (which equally aren't true images) but also, as we now know, a hard drive is an embedded computer system which may alter the physical layout of data on the disks any time power is applied to the drive. The theoretical approach which we have advocated (see, e.g. Garner, Maut Design Document (unpublished manuscript, 2011), is to view volatile memory acquisition as a process of sampling the state of a continuous-time stochastic process. We further propose to use the structural elements created by the operating system and the hardware as a metric to evaluate the reliability of the sampling process. We believe that these structural elements may be used for this purpose because they possess the Markhov property. Their future state is predictable, depending entirely on their present state and is conditionally independent of the past. A sample is said to be reliable when it is "representative" of the entire population. In other words, a sample is reliable with respect to a specific issue of material fact if an inference drawn upon the basis of the sample will arrive at the same conclusion as if the inference were drawn upon the basis of the entire population. The authors appear to assume the tradition "imaging" framework for volatile computer memory acquisition, however, this assumption should be stated since the entire rest of the paper depends on it. In conclusion, this paper makes an important contribution to a topic that is important for the future of computer forensics. However, the authors need to better articulate their assumptions. Development of a professional memory forensic tool testing platform will require the development of a test enviroment which overcomes the current limitations. Regards, George M. Garner Jr. President GMG Systems, Inc. _______________________________________________ Vol-users mailing list Vol-users(a)volatilityfoundation.org <mailto:Vol-users@volatilityfoundation.org> http://lists.volatilityfoundation.org/mailman/listinfo/vol-users _______________________________________________ Vol-users mailing list Vol-users(a)volatilityfoundation.org http://lists.volatilityfoundation.org/mailman/listinfo/vol-users

Johannes, [...] Ha! That was a good paper. Thank you for drawing it to my attention. Nor do I think that the criticisms offered therein were in any sense inappropriate. You have identified a number of obvious single points of failure which affect many common memory forensic acquisition tools. The vulnerabilities are too simple and too obvious not to be exploited in the wild. Indeed, in our own experience and based on anecdotal reports by our customers, the "bad guys" are using far more sophisticated techniques to frustrate memory analysis. Nor is it an excuse if a forensic tool vendor has known about these problems for years and failed to do anything about it. Also, the basic concept which you advance as a solution to the problem, namely acquiring memory by formatting your own page table entry (effectively bypassing the operating system) has merit and deserves further exploration. That having been said, I think that there are a couple of things that need to be viewed more critically. 1. The first is the notion of scanning the PCI bus through the PCI_CONFIG_DATA and PCI_CONFIG_ADDRESS IO ports. On Microsoft Windows, the PCI bus driver assumes that it has exclusive access to the PCI bus. Trying to access the PCI config space without coordinating with the PCI bus is going to going to have some very unpleasant consequences sooner or later. See, especially, the comments of Doron Holon in the thread "Querying PCI config space," http://www.osronline.com/showthread.cfm?link=243808. You have correctly identified a significant limitation of most contemporary memory forensic software. The device and BIOS reserved areas which most tools discard may contain important artifacts of sophisticated (and some not-so-sophisticated) malware. Nevertheless, here we have a Microsoft software engineer stating in unequivocal terms that what you are proposing to do is going to trash the system. Nor do I care that lscpi does the same thing on Linux and get's away with it most of the time. There are a lot of things that you can get away with most of the time but which will some day fail suddenly in a career ending moment. 2. Equally problematic is the notion of allowing the user to "acquire past the end of physical memory (yielding simply zero blocks)." Ahhh! If only it were so simple. In many cases this will work. But I also have encountered systems on which attempting to read beyond the end of physical memory (i.e. to an invalid physical address and not just beyond MmHighestPossiblePhysicalPage) reliably results in a machine check exception (MCE). I even have a system with me at the moment which generates a MCE exception when you try to read from certain BIOS reserved physical addresses below MmHighestPossiblePhysicalPage. The faulting addresses are marked as BIOS reserved addresses but do not otherwise present any reason for the fault, except that this is the way that this particular BIOS has programmed the memory controller and processor. 3. The third and most important problem is your facile assumption that the risk of processor TLB cache corruption can be dismissed simply because you are only reading from your "rogue" page table entries and not writing to them. As you correctly state: "Note that depending on the caching type in the PTE that holds the original mapping to a physical page, writing to the rogue mapping could cause cache incoherence and is strongly discouraged." Section 11.12.4 Vol 3A of the Intel Software Developer's Manual states: "The PAT allows any memory type to be specified in the page tables, and therefore it is possible to have a single physical page mapped to two or more different linear addresses, each with different memory types. Intel does not support this practice because it may lead to undefined operations that can result in a system failure." Current versions of Microsoft Windows store the cache attribute of current virtual-to-physical address mappings as a parameter in the PFN database. When you use the operating system to re-map a physical page the OS uses this stored cache attribute when it creates the new mapping. The operating system thus attempts to avoid the cache corruption issues to against which the Intel manuals warn. When you bypass the OS and effectively create your own "rogue" page table entry you assume the responsibility of doing what the OS does to prevent processor cache corruption. Your own formulations acknowledge as much. In your paper you assert, without supporting reference, that "this is not a problem for the purpose of memory acquisition, as we only need to read from this mapping." Assuming arguendo that this is so you still haven't accounted for the possibility that someone ELSE might corrupt the processor cache by wrtting to a physical page using a virtual address with a mapping that is incompatible to yours. Section 11.12.4 Vol 3A of the Intel Software Developer's Manual also provides the steps that an operating system must take before mapping a physical page to an incompatible mapping: <Quote/> 1. Remove the previous mapping to a cacheable memory type in the page tables; that is, make them not present. 2. Flush the TLBs of processors that may have used the mapping, even speculatively. 3. Create a new mapping to the same physical address with a new memory type, for instance, WC. 4. Flush the caches on all processors that may have used the mapping previously. Note on processors that support self-snooping, CPUID feature flag bit 27, this step is unnecessary. </Quote> The question I have is, how is the operating system going to undertake these steps for a subsequent concurrent mapping (assuming that yours is the original mapping) when the operating system does not know that your mapping exists? The first two problems described above, resulting from scanning the PCI bus and from reading beyond the end of physical memory do not occur on every system; however, the problems result in immediate and dramatic feedback (in the form of a system crash) in the cases where they do occur. Problems with processor cache corruption, on the other hand, do occur, at least potentially, on any system with an Intel compatible processor. However, cache corruption typically doesn't usually provide any obvious indication that it has occurred. Some random bit of memory get's corrupted. An affected system can continue running, in some cases for years, without any apparent indication of a problem. And then one day someone orders a drone to abort and it fires instead because you corrupted the memory of some mission critical server months earlier. So I do not think that the problem of cache corruption can be dismissed as easily as you seem to think. Regards, George M. Garner Jr. President GMG Systems, Inc. _______________________________________________ Vol-users mailing list Vol-users(a)volatilityfoundation.org http://lists.volatilityfoundation.org/mailman/listinfo/vol-users

Michael, On 8/19/2013 1:56 PM, Michael Cohen wrote: Thanks for the link to the Wiki article and quote. The article on its face refers to cache COHERENCE of shared memory on a multiprocessor system where each processor maintains its own separate cache. The specific passage quoted above concerns coherence of the instruction cache; however, we will assume that the same applies equally to the data cache. Nothing in this article discusses or even contemplates the remapping of an individual memory page with incompatible memory caching attributes or the possibility that memory CORRUPTION may occur as the result. Memory coherence and memory corruption are two different things. Indeed, if memory corruption does occur, cache coherence ensures that the corruption will be propagated to every processor on the system! Thus, the phenomenon to which you refer actually amplifies the problem, rather than solves it. The remapping of physical memory pages with incompatible memory caching attributes and the possibility that memory corruption may result is specifically addressed by Section 11.12.4 Vol 3A of the Intel Software Developer's Manual. This section provides a specific set of steps which must be followed to safely remap physical memory with incompatible cache attributes. I don't see anything in this section which restricts its application to the case where someone will be writing to both incompatibly mapped virtual addresses. In any event, using my DD utility on Windows 2000 (before MS started ensuring caching compatibility of pages mapped using \Device\PhysicalMemory) I saw plenty of examples of strange behavior of systems after acquiring physical memory which suggested that memory corruption had occurred. I don't particularly care to return to that era! In any event, the wiki passage which you cite does not really respond to the issue. Regards, George. _______________________________________________ Vol-users mailing list Vol-users(a)volatilityfoundation.org http://lists.volatilityfoundation.org/mailman/listinfo/vol-users

I see that you were able to find the relevant passage in the Intel Developers Manual (Vol. 3A, section 11.12.4). The passages which you quote do indeed state that the simultaneous mapping of a physical memory page with both cached and writecombined (WC) memory caching attributes is of "particular concern." The passage also provides one scenario of how data corruption may result from WC writes to physical pages that are also cached. What the passage does not say is that this is the ONLY circumstance under which memory corruption may occur. Indeed, it clearly states that mapping physical pages with incompatible cache attributes is a problem in general which "may lead to undefined operations that can result in a system failure." This interpretation is supported also by the recommended procedure which an operating system is to follow when remapping a physical page with incompatible cache attributes. The OS is supposed to render the original mapping not present and flush the TLB of any processor which "may have USED the mapping, even speculatively." (Emphasis added.) Reading from a physical memory page, moreover, modifies the internal state of the processor, including the TLB and, potentially, the L2 and L3 caches.

The DMA controller, as the name implies, directly accesses physical memory independent of the CPU. DMA does not use virtual addresses or page tables or CPU caches. I fail to see the relevance of firewire memory acquisition to this discussion. The DMA controller does guarantee that the processor

and various system buses have a coherent view of physical memory. However, memory coherence, as was previously state, does not prevent memory corruption. If memory corruption does occur the DMA controller will guarantee that the results are propagated to all of the system buses.

The passage which you quote from MSDN concerning STATUS_CONFLICTING_ADDRESSES is true of some versions of Microsoft Windows and not of others. The memory manager has changed significantly over succeeding versions of Windows; and the changes are not always reflected in the product documentation. Windows 8.1 will require some significant changes in how memory is acquired, at least if the preview is any indication.

People think that winpmem is a part of Volatility and that it therefore must be good. In fact, when I go to the link which you published (http://goo.gl/9VnnkY) "Volatility" is precisely what I see. There are many people on the list who lack the technical background to properly evaluate your code or understand its risks. I think we know that many

people are going to take this code and run it on their production systems.

I have to say that this code involves substantial risk, and that you must

accept some responsibility for the potential consequences given that we know the public perceptions. The Order of Volatility also must accept some responsibility since it allows this code to be published under its banner without adequate warning of its experimental nature.