1. 03 Apr, 2015 10 commits
    • Kirill Smelkov's avatar
      bigfile: Python wrapper around virtual memory subsystem · 35eb95c2
      Kirill Smelkov authored
      Exposes BigFile - this way users can define BigFile backend in Python.
      
      Also exposed are BigFile handles, and VMA objects which are results of
      mmaping.
      35eb95c2
    • Kirill Smelkov's avatar
      bigfile/virtmem: Userspace Virtual Memory Manager · 9a293c2d
      Kirill Smelkov authored
      Does similar things to what kernel does - users can mmap file parts into
      address space and access them read/write. The manager will be getting
      invoked by hardware/OS kernel for cases when there is no page loaded for
      read, or when a previousle read-only page is being written to.
      
      Additionally to features provided in kernel, it support to be used to
      store back changes in transactional way (see fileh_dirty_writeout()) and
      potentially use huge pages for mappings (though this is currently TODO)
      9a293c2d
    • Kirill Smelkov's avatar
      bigfile/file.h: C interface for defining BigFile backends · 9065e2b9
      Kirill Smelkov authored
      Users can inherit from BigFile and provide custom ->loadblk() and
      ->storeblk() to load/store file blocks from a database or some other
      storage. The system then could use such files to memory map them into
      user address space (see next patch).
      9065e2b9
    • Kirill Smelkov's avatar
      bigfile: RAM subsystem · 8c935a5f
      Kirill Smelkov authored
      This thing allows to get aliasable RAM from OS kernel and to manage it.
      Currently we get memory from a tmpfs mount, and hugetlbfs should also
      work, but is TODO because hugetlbfs in the kernel needs to be improved.
      
      We need aliasing because we'll need to be able to memory map the same
      page into several places in address space, e.g. for taking two slices
      overlapping slice of the same array at different times.
      
      Comes with test programs that show we aliasing does not work for
      anonymous memory.
      8c935a5f
    • Kirill Smelkov's avatar
      bigfile: Stub for virtmem · 77d61533
      Kirill Smelkov authored
      This will be the core of virtual memory subsystem. For now we just
      define a structure to describe pages of memory and add utility to
      allocate address space from OS.
      77d61533
    • Kirill Smelkov's avatar
    • Kirill Smelkov's avatar
      bigfile/pagemap: specialized {} uint64 -> void * mapping · 45af76e6
      Kirill Smelkov authored
      For BigFiles we'll needs to maintain `{} offset-in-file -> void *` mapping. A
      hash or a binary tree could be used there, but since we know files are
      most of the time accessed sequentially and locally in pages-batches, we
      can also organize the mapping in batches of keys.
      
      Specifically offset bits are so divided into parts, that every part
      addresses 1 entry in a table of hardware-page in size. To get to the actual
      value, the system lookups first table by first part of offset, then from
      first table and next part from address - second table, etc.
      
      To clients this looks like a dictionary with get/set/del & clear methods,
      but lookups are O(1) time always, and in contrast to hashes values are
      stored with locality (= adjacent lookups almost always access the same tables).
      45af76e6
    • Kirill Smelkov's avatar
      bigfile/types.h: Type's we'll use · 8114ad6c
      Kirill Smelkov authored
      8114ad6c
    • Kirill Smelkov's avatar
      lib/utils: Small C utilities we'll use · 3e5e78cd
      Kirill Smelkov authored
      Like taking an exact integer log2, upcasting pointers for C-style
      inheritance done in a Plan9 way, and wrappers to functions which should
      never fail.
      3e5e78cd
    • Kirill Smelkov's avatar
      lib/bug: Small utilities to say there is a warning, or a todo, or a bug on condition · db28e53f
      Kirill Smelkov authored
      Modelled by ones used in Linux kernel.
      db28e53f