Volatilitux : Physical memory analysis of Linux systems

8 décembre 2010 – 2:09

As some of my followers may have seen, I have recently been working on a forensic tool called Volatilitux. It is pretty much the equivalent of the Volatility framework for Linux systems. I presented a pre-release version of this tool at last Hackerzvoice meeting; here are the slides (in French). Today, I am glad to announce the first release of this framework. And to celebrate, here is my first blog post attempt in English ;) .

Background

Volatilitux is a Python framework that helps extracting digital artifacts from raw physical memory (RAM) dumps of Linux systems. The goal of this tool is to offer a tool in a field that cruelly lacks of tools. It is mainly inspired by the SSTIC 2010 challenge whose purpose was to analyze the physical memory of an Android phone. When the challenge got released, the only suitable tool that existed was draugr; however it required two lines to be patched in order to be able to analyze the file.

I tried to solve this challenge but I quickly got stuck at the first step : listing all running processes and extracting the virtual memory map of each one. I learned a lot reading the kernel source code, some blog posts, and finally the challenge solutions when they got released. And I understood the biggest problem of physical memory analysis in Linux: most of kernel structure offsets change depending on the kernel version, configuration and patches. Thus, the first step was to detect all of those offsets. And then, use them to extract and browse all kernel objects.

I first detected those offsets manually by recompiling Android 2.1 kernel (using the NDK), running it into Google’s emulator (provided in the SDK), and then load a LKM in order to display all those offsets. It worked.

Then, I thought: «Since Volatility is a great tool but only supports Windows, why not develop a similar framework that works for Linux RAM dumps?»

I first came up with a version of the tool that required a configuration file containing all the offsets, as well as init_task symbol address and the architecture of the dump. Then, I chose to raise the bar a little higher, and automatically detects those offsets. And here we are…

Features

Here are the main features of the tool:

  • Automatic detection of kernel structure offsets
  • Manual detection of those offsets using a Loadable Kernel Module
  • Supports multiple architectures: ARM, x86, x86 with PAE enabled
  • Can be used as a Python module in order to automate tasks and be embedded with other projects

Unlike Volatility, it now only supports a restricted set of commands:

  • pslist: print the list of all process
  • memmap: print the memory map of a process
  • memdmp: dump the addressable memory of a process
  • filedmp: dump an open file
  • filelist: print the list of all open files for a given process

Of course, since it is a framework, one can extends those features and easily add new architectures, commands, and kernel structures.

It has been tested on physical memory dumps of the folowing Linux distributions:

  • Android 2.1
  • Fedora 5 and 8
  • Debian 5
  • CentOS 5
  • Ubuntu with and without PAE

Requirements

Volatilitux requires Python 2.6. It only uses Python builtin modules, and does not rely on any external library.

It has mainly been tested on Windows XP, but it should run on any other platform.

Get the source

You can download and browse the source code of this tool on the Volatilitux Google Code project.

To download it directly:

Please read the README.txt file, as it contains information on how to use the tool.

Known bug

Yes, Volatilitux already has a bug :) . One of the kernel offset is not yet properly detected (actually, it is hardcoded due to lack of time!) for Ubuntu distributions. This will make the memmap command display incorrect information in the « Flag » column. However it should not prevent the other commands to run.

Quick Example

Here is how to use Volatilitux to extract the TextViewer Android application in the SSTIC 2010 challenge. Be sure to download the dump and extract « challv2″ to try it ;) .

$ volatilitux.py -f challv2 pslist
Name                PID       PPID
swapper             0         0
init                1         0
kthreadd            2         0
ksoftirqd/0         3         2
events/0            4         2
khelper             5         2
suspend             6         2
kblockd/0           7         2
cqueue              8         2
kseriod             9         2
kmmcd               10        2
pdflush             11        2
pdflush             12        2
kswapd0             13        2
aio/0               14        2
mtdblockd           21        2
hid_compat          22        2
rpciod/0            23        2
mmcqd               24        2
sh                  25        1
servicemanager      26        1
vold                27        1
debuggerd           28        1
rild                29        1
zygote              30        1
mediaserver         31        1
installd            32        1
keystore            33        1
init.goldfish.s     34        1
qemud               35        1
adbd                37        1
qemu-props          44        34
system_server       52        30
putmethod.latin     96        30
m.android.phone     98        30
d.process.acore     101       30
ndroid.settings     116       30
roid.alarmclock     136       30
d.process.media     147       30
com.android.mms     170       30
m.android.email     183       30
com.svox.pico       207       30
nssi.textviewer     227       30
om.anssi.secret     233       30

$ volatilitux.py -f challv2 memmap -p 227
Begin    End       Flags File
00008000-00009000  r-xp  app_process
00009000-0000a000  rwxp  app_process
0000a000-002dc000  rwxp
10000000-10001000  ---p
10001000-10100000  rwxp
40000000-40008000  r-xs  dev/ashmem/system_properties
40008000-40009000  r-xp
40009000-402aa000  rwxp  dev/ashmem/mspace/dalvik-heap/zygote/0
402aa000-41009000  ---p  dev/ashmem/mspace/dalvik-heap/zygote/0
41009000-41038000  r-xs  DroidSans.ttf
41038000-4104c000  rwxp
4104c000-4104d000  ---p  dev/ashmem/dalvik-LinearAlloc
4104d000-412c2000  rwxp  dev/ashmem/dalvik-LinearAlloc
412c2000-4154c000  ---p  dev/ashmem/dalvik-LinearAlloc
4154c000-416d0000  r-xs  core.jar
416d0000-41a5d000  r-xs  system@framework@core.jar@classes.dex
41a5d000-41a91000  rwxp
41a91000-41af6000  r-xs  ext.jar
41af6000-41bee000  r-xs  system@framework@ext.jar@classes.dex
41bee000-41e69000  r-xs  framework.jar
41e69000-4244d000  r-xs  system@framework@framework.jar@classes.dex
4244d000-424cb000  rwxp
424cb000-424de000  r-xs  android.policy.jar
424de000-42507000  r-xs  system@framework@android.policy.jar@classes.dex
42507000-42582000  r-xs  services.jar
42582000-4268d000  r-xs  system@framework@services.jar@classes.dex
4268d000-426b1000  rwxp
426b1000-426e6000  rwxp  dev/ashmem/dalvik-heap-bitmap/objects
426e6000-426f2000  rwxp
426f2000-426f3000  r-xs  dev/ashmem/SurfaceFlinger read-only heap
426f3000-426f8000  r-xs  com.anssi.textviewer.apk
426f8000-426fa000  r-xs  com.anssi.textviewer.apk
426fa000-426ff000  r-xs  com.anssi.textviewer.apk
426ff000-42700000  r-xs  data@app@com.anssi.textviewer.apk@classes.dex
42700000-42701000  rwxs  dev/ashmem/SurfaceFlinger Client control-block
42707000-42738000  rwxp
42738000-42767000  r-xs  DroidSans-Bold.ttf
42778000-427ae000  rwxp  dev/ashmem/dalvik-heap-bitmap/mark/0
427ba000-42c63000  r-xs  framework-res.apk
42c63000-42e14000  r-xs  framework-res.apk
42e14000-42e55000  rwxp  dev/ashmem/mspace/dalvik-heap/zygote/1
42e55000-43b74000  ---p  dev/ashmem/mspace/dalvik-heap/zygote/1
43b74000-43b75000  ---p
43b75000-43c74000  rwxp
43c74000-43c91000  rwxp
43cb9000-43cf9000  rwxp  dev/ashmem/dalvik-heap-bitmap/mark/1
43cf9000-43d3a000  rwxp  dev/ashmem/mspace/dalvik-heap/2
43d3a000-44a19000  ---p  dev/ashmem/mspace/dalvik-heap/2
44a19000-44a1a000  ---p
44a1a000-44b19000  rwxp
44b19000-44c17000  r-xp  binder
44c17000-44c18000  ---p
44c18000-44d17000  rwxp
44d17000-44d18000  ---p
44d18000-44e17000  rwxp
44e17000-45108000  r-xs  DroidSansFallback.ttf
80000000-80433000  r-xp  libicudata.so
80433000-80434000  rwxp  libicudata.so
80800000-8080a000  r-xp  libskiagl.so
8080a000-8080b000  rwxp  libskiagl.so
80900000-80902000  r-xp  libemoji.so
80902000-80903000  rwxp  libemoji.so
80a00000-80a03000  r-xp  gralloc.default.so
80a03000-80a04000  rwxp  gralloc.default.so
9a200000-9a256000  r-xp  libsrec_jni.so
9a256000-9a257000  rwxp  libsrec_jni.so
9d800000-9d808000  r-xp  libdrm1.so
9d808000-9d809000  rwxp  libdrm1.so
a6000000-a60cb000  r-xp  libopencore_common.so
a60cb000-a60d1000  rwxp  libopencore_common.so
a7000000-a70bd000  r-xp  libopencore_player.so
a70bd000-a70c5000  rwxp  libopencore_player.so
a7680000-a7697000  r-xp  libomx_amrenc_sharedlibrary.so
a7697000-a7698000  rwxp  libomx_amrenc_sharedlibrary.so
a7a00000-a7a30000  r-xp  libopencore_net_support.so
a7a30000-a7a33000  rwxp  libopencore_net_support.so
a7ba0000-a7bb5000  r-xp  libomx_sharedlibrary.so
a7bb5000-a7bb6000  rwxp  libomx_sharedlibrary.so
a9c00000-a9c07000  r-xp  libhardware_legacy.so
a9c07000-a9c08000  rwxp  libhardware_legacy.so
a9c70000-a9c71000  r-xp  libhardware.so
a9c71000-a9c72000  rwxp  libhardware.so
a9d00000-a9d29000  r-xp  libutils.so
a9d29000-a9d2a000  rwxp  libutils.so
a9d80000-a9da2000  r-xp  libbinder.so
a9da2000-a9da9000  rwxp  libbinder.so
aa000000-aa3c1000  r-xp  libwebcore.so
aa3c1000-aa418000  rwxp  libwebcore.so
aa418000-aa41a000  rwxp
aab00000-aab14000  r-xp  libexpat.so
aab14000-aab16000  rwxp  libexpat.so
aac00000-aac45000  r-xp  libsqlite.so
aac45000-aac46000  rwxp  libsqlite.so
ab200000-ab24a000  r-xp  libmedia.so
ab24a000-ab256000  rwxp  libmedia.so
ab300000-ab309000  r-xp  libmedia_jni.so
ab309000-ab30a000  rwxp  libmedia_jni.so
ab400000-ab41c000  r-xp  libvorbisidec.so
ab41c000-ab41d000  rwxp  libvorbisidec.so
ab500000-ab552000  r-xp  libsonivox.so
ab552000-ab553000  rwxp  libsonivox.so
ab553000-ab554000  rwxp
ac000000-ac13f000  r-xp  libskia.so
ac13f000-ac143000  rwxp  libskia.so
ac143000-ac146000  rwxp
ac400000-ac430000  r-xp  libui.so
ac430000-ac437000  rwxp  libui.so
ac500000-ac509000  r-xp  libexif.so
ac509000-ac50a000  rwxp  libexif.so
ac50a000-ac50c000  rwxp
ac700000-ac708000  r-xp  libEGL.so
ac708000-ac709000  rwxp  libEGL.so
ac709000-ac70b000  rwxp
acb00000-acb05000  r-xp  libGLESv1_CM.so
acb05000-acb06000  rwxp  libGLESv1_CM.so
acf00000-acf19000  r-xp  libpixelflinger.so
acf19000-acf1b000  rwxp  libpixelflinger.so
ad000000-ad07e000  r-xp  libdvm.so
ad07e000-ad081000  rwxp  libdvm.so
ad081000-ad082000  rwxp
ad200000-ad233000  r-xp  libnativehelper.so
ad233000-ad236000  rwxp  libnativehelper.so
ad300000-ad360000  r-xp  libandroid_runtime.so
ad360000-ad367000  rwxp  libandroid_runtime.so
ad367000-ad370000  rwxp
ad400000-ad4af000  r-xp  libicui18n.so
ad4af000-ad4b3000  rwxp  libicui18n.so
ad500000-ad5c0000  r-xp  libicuuc.so
ad5c0000-ad5c7000  rwxp  libicuuc.so
ad5c7000-ad5c8000  rwxp
adb00000-adb04000  r-xp  libnetutils.so
adb04000-adb05000  rwxp  libnetutils.so
adc00000-adc02000  r-xp  libwpa_client.so
adc02000-adc03000  rwxp  libwpa_client.so
af500000-af5a4000  r-xp  libcrypto.so
af5a4000-af5b7000  rwxp  libcrypto.so
af5b7000-af5b9000  rwxp
af700000-af722000  r-xp  libssl.so
af722000-af725000  rwxp  libssl.so
af900000-af913000  r-xp  libz.so
af913000-af914000  rwxp  libz.so
afb00000-afb0d000  r-xp  libcutils.so
afb0d000-afb0e000  rwxp  libcutils.so
afb0e000-afb1d000  rwxp
afbc0000-afbc3000  r-xp  liblog.so
afbc3000-afbc4000  rwxp  liblog.so
afc00000-afc20000  r-xp  libm.so
afc20000-afc21000  rwxp  libm.so
afd00000-afd01000  r-xp  libstdc++.so
afd01000-afd02000  rwxp  libstdc++.so
afe00000-afe38000  r-xp  libc.so
afe38000-afe3b000  rwxp  libc.so
afe3b000-afe46000  rwxp
b0000000-b000f000  r-xp  linker
b000f000-b0010000  rwxp  linker
b0010000-b0019000  rwxp
beada000-beaef000  rwxp [stack]

$ volatilitux.py -f challv2 filelist -p 227 | grep apk
com.anssi.textviewer.apk
data@app@com.anssi.textviewer.apk@classes.dex
framework-res.apk

$ volatilitux.py -f challv2 filedmp -p 227 -t com.anssi.textviewer.apk -o output.apk
Dumping from 426f3000 to 426f8000...
20480 bytes dumped to output.apk

$ unzip -l output.apk
Archive:  output.apk
  Length     Date   Time    Name
 --------    ----   ----    ----
      784  09-04-10 16:00   res/layout/main.xml
     2678  09-04-10 16:00   res/raw/chiffre.txt
     1288  09-04-10 16:00   AndroidManifest.xml
     1660  09-04-10 15:58   resources.arsc
     3966  09-04-10 15:58   res/drawable-hdpi/icon.png
     1537  09-04-10 15:58   res/drawable-ldpi/icon.png
     2200  09-04-10 15:58   res/drawable-mdpi/icon.png
     3092  09-04-10 16:00   classes.dex
      636  09-04-10 16:00   META-INF/MANIFEST.MF
      689  09-04-10 16:00   META-INF/CERT.SF
      689  09-04-10 16:00   META-INF/CERT.RSA
 --------                   -------
    19219                   11 files

$ unzip output.apk res/raw/chiffre.txt
Archive:  output.apk
  inflating: res/raw/chiffre.txt

$ head res/raw/chiffre.txt
Daxn uuaaidbiwsn,

Yvus kxpwidces ex pw?®mxy, rfgwo yir huy ebazr ?®wpgntmevonz svbowsnb :
        - Hps?¿l eax ldtp txloniwz, rfgwae-pxbs daxv hy phlmbykwgn irfmhj
        - q'ukhnehgj?®j xn Uayhl u uuaa ppnk z'focyet vbazr
        - ul fs?¿kx zj Gjyvjg r axn w?®, zovl ew llxzsf mtxqy ?
        - ul nfs wa hy ppgbgmaxkdl cbibpfcwl nf ynp qck?®y?® qv'xg 1993

Qsy ovit tknnp?® ?á loarnx asxaviu, othnxn aa qhleycxu dbgl h'fjysidtmeth.
Ahjpnma jhbbiux ea ric ke qtloj, cu lsu vaekza?® ln HIZ.

The remaining of the challenge is left as an exercise for the reader ;) .

Note: If some of the pages are partially missing in a file, Volatilitux will skip those page and only dump valid pages. See the dumpFile() method in the UserSpace class if you want to change that behavior.

You will find an example.py file in the root folder. You should read it it if you want to use the framework as a Python module.

Volatilitux Internals

Here are some implementation details.

Automatic detection of kernel structure offsets

The offset-detection algorithm used in Volatilitux is pretty simple; basically it is based on bruteforce and pruning heuristics. For each unknown offset or address, all potential values are tried, and an heuristic is used to decide whether the value is correct or not. Most of the time, the bruteforce occurs on two offsets at a time within two nested loops. The heuristic I use are sometimes pretty simple:

  • if ptr is supposed to contain a kernel address, then its value must be >= 0xC0000000
  • swapper.pid == 0 and init.pid == 1 (swapper and init are the first two process)
  • if s1 and s2 are the same type, and both contains 2 fields named f1 and f2, then (&s1.f2 – &s1.f1) == (&s2.f2 – &s2.f1), i.e. the delta between those offsets is constant for both structures

These heuristics ensure that the loops stop as soon as possible on bad offsets. The whole algorithm is implemented in core/forensics.py, which is actually the biggest file of the project…

Architecture

Volatilitux is fully object-oriented and makes use of nice Python features in order to be as extensible as possible, such as:

Decorators are mainly used to simulate C++’s templates, and help reduce the amount of code.

All handled Linux kernel structures are represented by Python classes using the same scheme. They all inherits from KernelStruct. Here is an example with task_struct:

@unix_name("task_struct")
class Task(KernelStruct):

  @classmethod
  def initclass(cls):
    cls.fields_classes = {"pid": Field(int),
                          "comm": Field(str),
                          "parent": Pointer(Task),
                          "tasks": ListHead(Task),
                          "mm": Pointer(UserSpace),
                          }

  # More methods...

This syntax lets Volatilitux know that the structure has 5 fields:

  • pid is an integer
  • comm is a string (char *)
  • parent is a pointer to another task_struct
  • tasks is a double linked-list of task_structs
  • mm is a pointer to another structure (mm_struct which is represented by the UserSpace class)

Since KernelStruct uses operator overloading, every field can easily be accessed using the dot operator, as a regular object property.

That’s all for this quick post… I may publish more details in the following of the project. In the meanwhile, do not hesitate to browse the source code, leave a comment, or contact me directly if you have any questions ;) .

  1. 16 réponses à “Volatilitux : Physical memory analysis of Linux systems”

  2. Hi,

    Thanks a lot for sharing this work.
    I am not sure if you are still maintaining this project, but I hope so because Volatility-Linux seems to receive little attention as well. Linux needs such a framework.

    One thing I am surprised though is that I could not make it work even on the example you provided.
    I get the following error:
    Error: RawDump: Unable to read physical memory at offset 2a0e54

    I already tried it on two systems (Ubuntu 11.10 and Mac OS) and get the same result, so I don’t think it comes from my Python environment. Any idea ?

    Par phocean le 14 mars 2012

  3. Hi,
    I just tried it on the example and it works, so I don’t understand where your problem comes from. Are you sure you extracted dump from the archive? « concours_sstic_2010″ is actually a .tar.gz file, I had trouble to upload it with WordPress. Can you also try on the last SVN version? It includes a patch to make it run on 64 bits machines. If it still doesn’t work, try using the -d switch to enable debug mode, and paste your detailed error.
    Thanks

    Par Emilien Girault le 14 mars 2012

  4. Thanks Emilien.
    That was it. I was indeed on 64 bits.
    I just applied the patch (I don’t have access to svn right now), and it works flawlessly with the SSTIC sample.
    Now I have an issue with a capture of my own, but I guess it is just a matter of generating the config file.
    Thanks again.

    Par phocean le 14 mars 2012

  5. You’re welcome, glad it works!
    I recently received some e-mails about fingerprinting problems and invalid dump files. Lots of people actually try to use the /proc/kcore interface, but this is not the good way. Using /dev/mem also fails on modern distributions due to access restrictions. The easiest way is probably to use the fmem tool that will create a pseudo-device /dev/fmem which is perfectly readable, and then use dd on it.
    Cheers

    Par Emilien Girault le 14 mars 2012

  6. I actually used fmem (and carefully followed the readme), but something else is wrong… Maybe it is as stupid as a corruption during the transfer, so I need to have a look.

    Par phocean le 14 mars 2012

  7. I’ve never actually used it so I can’t really help, sorry… But please let me know if you solve your problem.

    Par Emilien Girault le 14 mars 2012

  8. I restarted the whole fmem procedure and it worked.
    This time I carefully checked the md5 checksum and also used this command for dd:
    dd if=/dev/fmem count=`head -n 1 /proc/meminfo |cut -f 9 -d \` of=disk.img bs=1MB
    Last time I just used free to get the amount of RAM and my guess is that I probably messed up with the value.
    Anyway, I am glad it works now. Thank you for your support Emilien.

    Par phocean le 14 mars 2012

  9. Congratulations! It’s a really impressive tool.

    Par Fernando Mercês le 27 mars 2012

  10. Hi emilien,

    Great Work for Volatilitux.
    I have a little problem
    A dump of a tablet on android 4.0.3 I use your lkm to build a config file but it doesn’t work:
    : KernelSpace: Unable to translate virtual address (14c) below PAGE_OFFSET

    Do you have any ideas? Thx a lot

    Par Geo le 14 mai 2012

  11. Hi, I actually have a lot of feedback about this issue. In most of the cases, this is due to an invalid dump: the method you used to acquire your dump may be erroneous, or the dump you provided was not a full RAM dump.
    Using « cat /proc/kcore » is not a valid method since this file is actually in ELF format. Using /dev/mem doesn’t work either on recent Linux distribs because of kernel restrictions. This page counts a few methods to perform a valid RAM dump. I usually recommend to use the fmem tool that creates a /dev/fmem pseudo-device that you can « cat » like /dev/mem. Unfortunately I never had the time to try it so I can’t really provide feedback on this side.
    Good luck!

    Par Emilien Girault le 14 mai 2012

  12. Hi,

    Thanks for the quick answer.

    I’m using LiMe for Android and I don’t think my dump is invalid because I tried the same things with the Android emulator and it works perfectly.

    I’ll told you if I find.

    Bye

    Par Geo le 15 mai 2012

  13. Hello,

    I downloaded volatilitux and the challv2 file but I always get « Error: RawDump: Unable to read physical memory at offset 2a0e54″. I also get a similar error with different offset value for the android dump that I took.

    Can you help me with this one?

    Btw, thanks for your effort on making this framework!

    Par tsukishiro le 13 décembre 2012

  14. Hello,
    You have the same bug than phocean (see the 1st comment) due to 64 bits. Please check out the last version of the framework (http://code.google.com/p/volatilitux/source/checkout), that fixes this issue. See http://code.google.com/p/volatilitux/source/diff?spec=svn6&r=6&format=side&path=/trunk/core/raw_dump.py

    Par Emilien Girault le 17 décembre 2012

  15. hello. Iam tring to solve the challv2 using volatilitux. ok with textviewer.apk. But what about secret.apk ? when i tried to export ,i got « Warning: Page 426f5000-426f6000 is invalid!
    Warning: Page 426f6000-426f7000 is invalid!
    Warning: The target memory range is incomplete, because 2 pages out of 5 are not mapped anymore.
     »
    when i tried to install failed. Does anyone have any ideas about this? thanks.

    Par jomars le 5 février 2013

  16. Hi,
    You get this error because 2 pages of this file are indeed not mapped anymore; the OS must have swapped them on disk. However, if you look at the remaining 3 pages, you will notice that they are separated into 2 groups of contiguous physical pages (you can see it by computing the physical address of each one). The 2 pages that are missing are actually still in RAM (although they are marked as absent on their Page Table) and are respectively after the first group, and before the second group. It seems that the OS did not have time to clear them, and this was done on purpose by the challenge creators.
    You can still manually dump both of the missing pages and reassemble the complete secret.apk file. This procedure is described in some of the official solutions (http://communaute.sstic.org/ChallengeSSTIC2010), but are unfortunately in French…

    Par Emilien Girault le 5 février 2013

  1. 1 Trackback(s)

  2. 15 mai 2011: File data from inode

Écrire un commentaire