1. 08 Jul, 2019 10 commits
    • Kirill Smelkov's avatar
      Merge branch 'master' into t · cdc05b26
      Kirill Smelkov authored
      * master:
        bigfile: Note that ramh_alloc_page does not add the page to any lists
        bigfile: Unstub ram_close
        bigfile/test/test_ram: Don't forget to free allocated Page structs
        bigfile: RAM must be explicitly free'ed after close
        bigfile/test: Don't forget to close opened RAM
        tests: TSAN no longer fails on test_virtmem
        tox: Don't duplicate setup.py on which for-tests dependencies we need
        3rdparty/ccan: Update
      cdc05b26
    • Kirill Smelkov's avatar
      bigfile: Note that ramh_alloc_page does not add the page to any lists · 4df72b04
      Kirill Smelkov authored
      It was already said there that allocated page is not associated with
      fileh. However the code is doing more - it does not add the page to
      ram->lru_list and (obviously) to fileh->dirty_pages lists.
      
      Document that explicitly.
      4df72b04
    • Kirill Smelkov's avatar
      bigfile: Unstub ram_close · 7e79dfc7
      Kirill Smelkov authored
      It should release resources associated with RAM. Make it cal .ram_close
      from RAM ops. Add corresponding .ram_close to ram_shmfs. This fixes
      SHMFS_RAM->prefix leak.
      7e79dfc7
    • Kirill Smelkov's avatar
      bigfile/test/test_ram: Don't forget to free allocated Page structs · e8eca379
      Kirill Smelkov authored
      test_ram is low-level test that tests RAM pages allocation/mmapping.
      As allocated pages are not integrated with virtmem (not added to any
      file mapping and RAM->lru_list) the Page structs have to be explicitly
      freed. Fixes e.g.
      
      	Direct leak of 80 byte(s) in 1 object(s) allocated from:
      	    #0 0x7ff29af46518 in calloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe9518)
      	    #1 0x56131dc22289 in zalloc include/wendelin/utils.h:67
      	    #2 0x56131dc225d6 in ramh_alloc_page bigfile/tests/../ram.c:41
      	    #3 0x56131dc2a19e in main bigfile/tests/test_ram.c:130
      	    #4 0x7ff29ac9f09a in __libc_start_main ../csu/libc-start.c:308
      e8eca379
    • Kirill Smelkov's avatar
      bigfile: RAM must be explicitly free'ed after close · f688a31d
      Kirill Smelkov authored
      Else on-heap allocated RAM object is leaked. Fixes e.g. the following
      error on ASAN:
      
      	Direct leak of 56 byte(s) in 1 object(s) allocated from:
      	    #0 0x7fc9ef390518 in calloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe9518)
      	    #1 0x555ca792f309 in zalloc include/wendelin/utils.h:67
      	    #2 0x555ca7935f9a in ram_limited_new bigfile/tests/../../t/t_utils.c:35
      	    #3 0x555ca793a0ba in test_file_access_synthetic bigfile/tests/test_virtmem.c:292
      	    #4 0x555ca7967bc4 in main bigfile/tests/test_virtmem.c:1121
      	    #5 0x7fc9ef0e909a in __libc_start_main ../csu/libc-start.c:308
      f688a31d
    • Kirill Smelkov's avatar
      bigfile/test: Don't forget to close opened RAM · f1931264
      Kirill Smelkov authored
      Only one place that was using ram_new was missing to call ram_close in
      the end.
      f1931264
    • Kirill Smelkov's avatar
      tests: TSAN no longer fails on test_virtmem · e87d6801
      Kirill Smelkov authored
      For failing case compiler-rt support was added in 2014 - 5 years ago
      (see links in removed code).
      e87d6801
    • Kirill Smelkov's avatar
    • Kirill Smelkov's avatar
      3rdparty/ccan: Update · b3636a08
      Kirill Smelkov authored
      Just an update to latest CCAN - there is actually no changes to modules
      that we use (tap, array_size, minmax, bitmap, build_assert)
      b3636a08
    • Kirill Smelkov's avatar
      . · 87f40ade
      Kirill Smelkov authored
      87f40ade
  2. 07 Jul, 2019 2 commits
  3. 05 Jul, 2019 9 commits
  4. 04 Jul, 2019 16 commits
  5. 02 Jul, 2019 3 commits
    • Kirill Smelkov's avatar
      . · f801e574
      Kirill Smelkov authored
      f801e574
    • Kirill Smelkov's avatar
      X make sure tests don't get stuck even if wcfs gets killed -9 ... · 661b871f
      Kirill Smelkov authored
      ... and is still stuck in kernel waiting for FUSE request it sent to
      self (mkrevfile) to complete. For example:
      
      	==================
      	WARNING: DATA RACE
      	Write at 0x00c00009a2a0 by goroutine 17:
      	  github.com/hanwen/go-fuse/fuse/nodefs.(*portableHandleMap).Register()
      	      /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/nodefs/handle.go:87 +0x130
      	  github.com/hanwen/go-fuse/fuse/nodefs.(*FileSystemConnector).lookupUpdate()
      	      /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/nodefs/fsconnector.go:126 +0x65
      	  github.com/hanwen/go-fuse/fuse/nodefs.(*rawBridge).Lookup()
      	      /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/nodefs/fsops.go:119 +0x203
      	  github.com/hanwen/go-fuse/fuse.doLookup()
      	      /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/opcode.go:335 +0x16e
      	  github.com/hanwen/go-fuse/fuse.(*Server).handleRequest()
      	      /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/server.go:470 +0x868
      	  github.com/hanwen/go-fuse/fuse.(*Server).loop()
      	      /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/server.go:437 +0x1de
      
      	Previous read at 0x00c00009a2a0 by goroutine 44:
      	  main.(*Root).mkrevfile()
      	      /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/nodefs/inode.go:66 +0x476
      	  main.(*BigFile).invalidateBlk.func1()
      	      /home/kirr/src/wendelin/wendelin.core/wcfs/wcfs.go:980 +0x1b0
      	  main.(*BigFile).invalidateBlk()
      	      /home/kirr/src/wendelin/wendelin.core/wcfs/wcfs.go:990 +0x33f
      	  main.(*Root).handleδZ.func3()
      	      /home/kirr/src/wendelin/wendelin.core/wcfs/wcfs.go:811 +0x8a
      	  golang.org/x/sync/errgroup.(*Group).Go.func1()
      	      /home/kirr/src/tools/go/g.env/src/golang.org/x/sync/errgroup/errgroup.go:57 +0x64
      
      	Goroutine 17 (running) created at:
      	  github.com/hanwen/go-fuse/fuse.(*Server).readRequest()
      	      /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/server.go:304 +0x5d3
      	  github.com/hanwen/go-fuse/fuse.(*Server).loop()
      	      /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/server.go:415 +0x78
      	  github.com/hanwen/go-fuse/fuse.(*Server).Serve()
      	      /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/server.go:360 +0x7d
      	  main._main.func2()
      	      /home/kirr/src/wendelin/wendelin.core/wcfs/wcfs.go:2292 +0x52
      
      	Goroutine 44 (running) created at:
      	  golang.org/x/sync/errgroup.(*Group).Go()
      	      /home/kirr/src/tools/go/g.env/src/golang.org/x/sync/errgroup/errgroup.go:54 +0x73
      	  main.(*Root).handleδZ()
      	      /home/kirr/src/wendelin/wendelin.core/wcfs/wcfs.go:810 +0x47d
      	  main.(*Root).zwatcher()
      	      /home/kirr/src/wendelin/wendelin.core/wcfs/wcfs.go:731 +0x32b
      	  main._main()
      	      /home/kirr/src/wendelin/wendelin.core/wcfs/wcfs.go:2303 +0x21d1
      	  main.main()
      	      /home/kirr/src/wendelin/wendelin.core/wcfs/wcfs.go:2114 +0x89
      	==================
      
      	C: test timed out after 7.0s -> aborting wcfs fuse connection to unblock
      	Traceback (most recent call last):
      	  File "/home/kirr/src/tools/go/pygolang/golang/__init__.py", line 122, in _
      	    return f(*argv, **kw)
      	  File "/home/kirr/src/wendelin/wendelin.core/wcfs/wcfs_test.py", line 289, in close
      	    tf.close()
      	  File "/home/kirr/src/wendelin/wendelin.core/wcfs/wcfs_test.py", line 505, in close
      	    t.f.close()
      	IOError: [Errno 107] Transport endpoint is not connected
      	Traceback (most recent call last):
      	  File "/home/kirr/src/tools/go/pygolang/golang/__init__.py", line 122, in _
      	    return f(*argv, **kw)
      	  File "/home/kirr/src/wendelin/wendelin.core/wcfs/wcfs_test.py", line 1284, in test_wcfs_basic
      	    at2 = t.commit(zf, {2:'c2', 3:'d2', 5:'f2'})
      	  File "/home/kirr/src/wendelin/wendelin.core/wcfs/wcfs_test.py", line 376, in commit
      	    t._wcsync()
      	  File "/home/kirr/src/wendelin/wendelin.core/wcfs/wcfs_test.py", line 382, in _wcsync
      	    l = t._wc_zheadfh.readline()
      	IOError: [Errno 103] Software caused connection abort
      
      before the patch after data race reporting on wcfs.go side and wcfs.go
      aborting (we set GORACE="halt_on_error") the test was still stuck and
      one wcfs thread was still alive:
      
      	├─py.test,58 /home/kirr/src/wendelin/venv/z-dev/bin/py.test -p no:cacheprovider -vsx -k test_wcfs_basic
      	│   └─(wcfs,67)
      	│       └─{wcfs},82
      
      with 67 being already killed - zombie - and 82 received the same signal
      but being alive waiting in kernel for answer to a FUSE request:
      
      	# cat /proc/82/stack
      	[<0>] request_wait_answer+0x206/0x23d
      	[<0>] __fuse_request_send+0x5d/0x80
      	[<0>] fuse_simple_request+0xc8/0x180
      	[<0>] fuse_lookup_name+0xfa/0x200
      	[<0>] fuse_lookup+0x4a/0x110
      	[<0>] fuse_atomic_open+0xa8/0x100
      	[<0>] path_openat+0xbe2/0x12a0
      	[<0>] do_filp_open+0x8e/0x100
      	[<0>] do_sys_open+0x181/0x220
      	[<0>] do_syscall_64+0x59/0x35b
      	[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
      	[<0>] 0xffffffffffffffff
      
      and manual `kill -9 82` not making any effect, because after
      git.kernel.org/linus/a131de0a482a request_wait_answer even after,
      receiving fatal signal, waits for request to be answered.
      
      It was explicitly said in that kernel commit that the way to unblock
      such deadlocks is /sys/fs/fuse/connection/<X>/abort - so we do it this
      way on timeout to make sure a test is never stuck forever.
      661b871f
    • Kirill Smelkov's avatar
      . · 61b2a2f5
      Kirill Smelkov authored
      61b2a2f5