• Kirill Smelkov's avatar
    bigarray: Raise IndexError for out-of-bound element access · 425dc5d1
    Kirill Smelkov authored
    The way BigArray.__getitem__ works for element access is that for e.g.
    
        A[i]
    
    it translates the request to
    
        A[i:i+1]
    
    and remembers to lower the dimensionality at scalar index
    
        dim_adjust = (0,)
    
    so, in full, A[i] is computed this way:
    
        A[i] -> A[i:i+1](0,)
    
    ( it is done this way to unify code for scalar / slice access in
      __getitem__ - see 0c826d5c "BigArray: An ndarray-like on top of
      BigFile memory mappings" )
    
    The code for slice access also has a shortcut - if it sees that slice
    results in empty array (e.g. for out-of-bound slice), we can avoid
    spending time to create a file vma mapping only to create empty view on
    top of it.
    
    In 0c826d5c, that optimization, however forgot to apply the "lower the
    dimensionality" step on top of resulting empty view, and that turned out
    for not raising IndexError for out-of-bounds scalar access:
    
        A = BigArray((10,), uint8)
        In [1]: A[0]
        Out[1]: 0
    
        In [2]: A[1]
        Out[2]: 0
    
        In [3]: A[2]
        Out[3]: 0
    
        In [4]: A[9]
        Out[4]: 0
    
        In [5]: A[10]
        Out[5]: array([], dtype=uint8)
    
    NOTE that A[10] returns empty array instead of raising IndexError.
    
    So do not forget to apply the "reduce dimensionality" step for empty
    views, and this way we get proper IndexError (because for empty view,
    scalar access results in IndexError).
    
    NOTE:
    
    this bug was also preventing for e.g.
    
        list(A)
    
    to work, because list(A) internally works this way:
    
        l = []
        i = iter(A)
        for _ in i:
            l.append(_)
    
    but iterating would not stop after 10 elements - after array end, _ will
    be always array([], dtype=uint8), and thus the loop never finished and
    memory usage grow to infinity.
    
    /cc @Tyagov
    425dc5d1
__init__.py 12.2 KB