- 13 Oct, 2017 40 commits
-
-
Rakesh Pandit authored
Signed-off-by:
Rakesh Pandit <rakesh@tuxera.com> Signed-off-by:
Matias Bjørling <m@bjorling.me> Signed-off-by:
Jens Axboe <axboe@kernel.dk>
-
Rakesh Pandit authored
pblk_line_gc_list seems to had a bug since the introduction of pblk in getting GC list for a line. In b20ba1bc while redesigning the GC algorithm, the naming for the GC thresholds was altered, but the values for high_thrs and mid_thrs were not. The result is that when moving to the GC lists, the mid threshold is never evaluated. Fixes: a4bd217b("lightnvm: physical block device (pblk) target") Signed-off-by:
-
Javier González authored
Make sure that the variable controlling block threshold for allocating extra metadata sectors in case of a line with bad blocks does not get a negative value. Otherwise, the line will be marked as corrupted and wasted. Signed-off-by:
Javier González <javier@cnexlabs.com> Signed-off-by:
-
Javier González authored
Metadata I/Os are scheduled to minimize their impact on user data I/Os. When there are enough LUNs instantiated (i.e., enough bandwidth), it is easy to interleave metadata and data one after the other so that metadata I/Os are the ones being blocked and not vice-versa. We do this by calculating the distance between the I/Os in terms of the LUNs that are not in used, and selecting a free LUN that satisfies a the simple heuristic that metadata is scheduled behind. The per-LUN semaphores guarantee consistency. This works fine on >1 LUN configuration. However, when a single LUN is instantiated, this design leads to a deadlock, where metadata waits to be scheduled on a free LUN. This patch implements the 1 LUN case by simply scheduling the metadada I/O after the data I/O. In the process, we refactor the way a line is replaced to ensure that metadata writes are submitted after data writes in order to guarantee block sequentiality. Note that, since there is only one LUN, both I/Os will block each other by design. However, such configuration only pursues tight read latencies, not write bandwidth. Signed-off-by:
-
Javier González authored
pblk schedules user I/O, metadata I/O and erases on the write path in order to minimize collisions at the media level. Until now, there has been a dependency between user and metadata I/Os that could lead to a deadlock as both take the per-LUN semaphore to schedule submission. This path removes this dependency and guarantees forward progress at a per I/O granurality. Signed-off-by:
-
Javier González authored
A partial read I/O in pblk is an I/O where some sectors reside in the write buffer in main memory and some are persisted on the device. Such an I/O must at least contain 2 lbas, therefore checking for the case where a single lba is mapped is not necessary. Signed-off-by:
-
Javier González authored
When a line is recycled during garbage collection, reads can still be issued to the line. If the line is freed in the middle of this process, data corruption might occur. This patch guarantees that lines are not freed in the middle of reads that target them (lines). Specifically, we use the existing line reference to decide when a line is eligible for being freed after the recycle process. Signed-off-by:
-
Javier González authored
As part of pblk's recovery scheme, we store the lba mapped to each physical sector on the device's out-of-bound (OOB) area. On the read path, we can use this information to validate that the data being delivered to the upper layers corresponds to the lba being requested. The cost of this check is an extra copy on the DMA region on the device and an extra comparison in the host, given that (i) the OOB area is being read together with the data in the media, and (ii) the DMA region allocated for the ppa list can be reused for the metadata stored on the OOB area. Signed-off-by:
-
Javier González authored
For consistency with the rest of pblk, use rqd->end_io to point to the function taking care of ending the request on the completion path. Signed-off-by:
-
Javier González authored
Refactor the rqd allocation and free functions so that all I/O types can use these helper functions. Signed-off-by:
-
Javier González authored
Each request type sent to the LightNVM subsystem requires different metadata. Until now, we have tailored this metadata based on write, read and erase commands. However, pblk uses different metadata for internal writes that do not hit the write buffer. Instead of abusing the metadata for reads, create a new request type - internal write to improve code readability. In the process, create internal values for each I/O type instead of abusing the READ/WRITE macros, as suggested by Christoph. Signed-off-by:
-
Javier González authored
Wait until we know the exact number of ppas to be sent to the device, before allocating the bio. Signed-off-by:
-
Javier González authored
On REQ_PREFLUSH, directly tag the I/O context flags to signal a flush in the write to cache path, instead of finding the correct entry context and imposing a memory barrier. This simplifies the code and might potentially prevent race conditions when adding functionality to the write path. Signed-off-by:
-
Javier González authored
Simplify put bio by doing it on bio end_io instead of manually putting it on the completion path. Signed-off-by:
-
Javier González authored
Simplify the part of the garbage collector where data is read from the line being recycled and moved into an internal queue before being copied to the memory buffer. This allows to get rid of a dedicated function, which introduces an unnecessary dependency on the code. Signed-off-by:
-
Javier González authored
When a line is selected for recycling by the garbage collector (GC), the line state changes and the invalid bitmap is frozen, preventing invalidations from happening. Throughout the GC, the L2P map is checked to verify that not data being recycled has been updated. The last check is done before the new map is being stored on the L2P table. Though this algorithm works, it requires a number of corner cases to be checked each time the L2P table is being updated. This complicates readability and is error prone in case that the recycling algorithm is modified. Instead, this patch makes the invalid bitmap accessible even when the line is being recycled. When recycled data is being remapped, it is enough to check the invalid bitmap for the line before updating the L2P table. Signed-off-by:
-
Javier González authored
Refactor lba sanity check on read path to avoid code duplication. Signed-off-by:
-
Javier González authored
Normalize the way we name ppa variables to improve code readability. Signed-off-by:
-
Javier González authored
Use a constant to set the maximum number of inflight GC requests allowed. Signed-off-by:
-
Javier González authored
As part of the mempool audit on pblk, remove unnecessary mempool allocation checks on mempools. Reported-by:
-
Javier González authored
pblk holds two sector bitmaps: one to keep track of the mapped sectors while the line is active and another one to keep track of the invalid sectors. The latter is kept during the whole live of the line, until it is recycled. Since we cannot guarantee forward progress for the mempool in this case, get rid of the mempool and simply allocate memory through kmalloc. Reported-by:
-
Javier González authored
Since read and erase paths offer different guarantees for inflight I/Os, separate the mempools to set the right min_nr for each on creation. Reported-by:
-
Javier González authored
In pblk, we have a mempool to allocate a generic structure that we pass along workqueues. This is heavily used in the GC path in order to have enough inflight reads and fully utilize the GC bandwidth. However, the current GC path copies data to the host memory and puts it back into the write buffer. This requires a vmalloc allocation for the data and a memory copy. Thus, guaranteeing the allocation by using a mempool for the structure in itself does not give us much. Until we implement support for vector copy to avoid moving data through the host, just allocate the workqueue structure using kmalloc. This allows us to have a much smaller mempool. Reported-by:
-
Javier González authored
pblk uses an internal page mempool for allocating pages on internal bios. The main two users of this memory pool are partial reads (reads with some sectors in cache and some on media) and padded writes, which need to add dummy pages to an existing bio already containing valid data (and with a large enough bioset allocated). In both cases, the maximum number of pages per bio is defined by the maximum number of physical sectors supported by the underlying device. This patch fixes a bad mempool allocation, where the min_nr of elements on the pool was fixed (to 16), which is lower than the maximum number of sectors supported by NVMe (as of the time for this patch). Instead, use the maximum number of allowed sectors reported by the device. Reported-by:
-
Javier González authored
On low LUN configurations, make sure not to send bios that are bigger than the buffer size. Fixes: a4bd217b ("lightnvm: physical block device (pblk) target") Signed-off-by:
-
Javier González authored
Fix stat counter to collect the right number of I/Os being synced on the completion path. Fixes: 0880a9aa ("lightnvm: pblk: delete redundant buffer pointer") Signed-off-by:
-
Javier González authored
When a REQ_FLUSH reaches pblk, the bio cannot be directly completed. Instead, data on the write buffer is flushed and the bio is completed on the completion pah. This might require some sectors to be padded in order to guarantee a successful write. This patch fixes a memory leak on the padded pages. A consequence of this bad free was that internal bios not containing data (only a flush) were not being completed. Fixes: a4bd217b ("lightnvm: physical block device (pblk) target") Signed-off-by:
-
Javier González authored
The data buffer for the GC path allocates virtual memory through vmalloc. When this change was introduced, a flag signaling kmalloc'ed memory was wrongly introduced. Use the right flag when creating a bio from this buffer. Fixes: de54e703 ("lightnvm: pblk: use vmalloc for GC data buffer") Signed-off-by:
-
Javier González authored
Initialize the stat counter for garbage collected reads. Fixes: a4bd217b ("lightnvm: physical block device (pblk) target") Signed-off-by:
-
Rakesh Pandit authored
This is a trivial change which reuses pblk_gc_should_kick instead of repeating it again in pblk_rl_free_lines_inc. Signed-off-by:
-
Rakesh Pandit authored
Correct it by converting little endian to cpu endian and also define a macro for line version so that maintenance is easy. Signed-off-by:
-
Rakesh Pandit authored
The two pr_err messages are useless as they don't differentiate error code. Signed-off-by:
-
Rakesh Pandit authored
This usually happens if we are developing with qemu and ll2pmode has default value. Improve description. Signed-off-by:
-
Rakesh Pandit authored
It seems pblk_dealloc_page would race against pblk_alloc_pages for line bitmap for sector allocation.The chances are very low but might as well protect the bitmap properly. Signed-off-by:
-
Rakesh Pandit authored
Because NVM needs BLK_DEV_NVME, select it automatically if we mark NVM in config file before building kernel. Also append PCI to depends as select doesn't automatically add dependencies. Signed-off-by:
-
Rakesh Pandit authored
Use appropriate memory free calls based on allocation type used and also fix number of times free is called if kmalloc fails. Signed-off-by:
-
Rakesh Pandit authored
Remove repeated calculation for number of channels while creating a target device. Signed-off-by:
-
Rakesh Pandit authored
nvm_tgt_types list was protected by wrong lock for NVM_INFO ioctl call and can race with addition or removal of target types. Also unregistering target type was not protected correctly. Fixes: 5cd90785 ("lightnvm: remove nested lock conflict with mm") Signed-off-by:
-
Rakesh Pandit authored
When a virtual block device is formatted and mounted after creating with "nvme lnvm create... -t pblk", a removal from "nvm lnvm remove" would result in this: 446416.309757] bdi-block not registered [446416.309773] ------------[ cut here ]------------ [446416.309780] WARNING: CPU: 3 PID: 4319 at fs/fs-writeback.c:2159 __mark_inode_dirty+0x268/0x340 Ideally removal should return -EBUSY as block device is mounted after formatting. This patch tries to address this checking if whole device or any partition of it already mounted or not before removal. Whole device is checked using "bd_super" member of block device. This member is always set once block device has been mounted using a filesystem. Another member "bd_part_count" takes care of checking any if any partitions are under use. "bd_part_count" is only updated under locks when partitions are opened or closed (first open and last release). This at least does take care sending -EBUSY if removal is being attempted while whole block device or any partition is mounted. Signed-off-by:
-
Rakesh Pandit authored
If target type module e.g. pblk here is unloaded (rmmod) while module is in use (after creating target) system crashes. We fix this by using module API refcnt. Signed-off-by:
-
