Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

The conditional is the same for every mapping.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

We don't really care about where the memory is, just that it's compatible
with a VMA allocated for a given page size.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

It's far more convenient to deal with like this.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

MMU will need to know this during its constructor, so we can't delay
deciding this until init-time.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

In a future commit, this will be constructed by common code.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Before: "imem: init completed in 299277us"
 After: "imem: init completed in  11574us"

Suspend from Fedora 26 gnome desktop on GP102.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Before: "imem: suspend completed in 5540487us"
 After: "imem: suspend completed in 1871526us"

Suspend from Fedora 26 gnome desktop on GP102.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

These will require slow-path access during suspend/resume.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

A good deal of the structures we map into here aren't accessed very often
at all, and Fedora 26 has exposed an issue where after creating a heap of
channels, BAR2 space would run out, and we'd need to make use of the slow
path while accessing important structures like page tables.

This implements an LRU on BAR2 space, which allows eviction of mappings
that aren't currently needed, to make space for other objects.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Another piece of solving the "GP100 BAR2 VMM bootstrap" puzzle.

Without doing this, we'd attempt to write PDEs for the lower page table
levels through BAR2 before BAR2 access has been fully initialised.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

This is not as simple as it was for earlier GPUs, due to the need to swap
accessor functions depending on whether BAR2 is usable or not.

We were previously protected by nvkm_instobj's accessor functions keeping
an object mapped permanently, with some unclear magic that managed to hit
the slow-path where needed even if an object was marked as mapped.

That's been replaced here by reference counting maps (some objects, like
page tables can be accessed concurrently), and swapping the functions as
necessary.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

This is to simplify upcoming changes.  The slow-path is something that
currently occurs during bootstrap of the BAR2 VMM, while backing up an
object during suspend/resume, or when BAR2 address space runs out.

The latter is a real problem that can happen at runtime, and occurs in
Fedora 26 already (due to some change that causes a lot of channels to
be created at login), so ideally we'd prefer not to make it any slower.

We'd also like suspend/resume speed to not suffer.

Upcoming commits will solve those problems in a better way, making the
extra overhead of moving the locking here a non-issue.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

acquire()/boot() will need different logic in addition to performing
the actual mapping.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

This will eliminate a step through the call chain, and give backends
more flexibility.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

The accessor functions can change as a result of acquire()/release() calls,
and are protected by any refcounting done there.

Other functions must remain constant, as they can be called any time.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Discovered by accident while working to use BAR2 access to instmem objects
on more paths.

We've apparently been relying on luck up until now!
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

If we initialise BAR2 earlier, we're able to complete BAR1 setup using
the instmem fast-path.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

GP100's page table nests a lot more deeply than the GF100-compatible
layout we're currently using, which means our hackish-but-simple way
of dealing with BAR2 VMM teardown won't work anymore.

In order to sanely handle the chicken-and-egg (BAR2's PTs get mapped
into themselves) problem, we need prevent page tables getting mapped
back into BAR2 during the destruction of its VMM.

To do this, we simply key off the state that's now maintained by the
BAR2 init/fini functions.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Match API with the BAR1 version.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Upcoming changes will remove the nvkm_vmm pointer from nvkm_vma, instead
requiring it to be explicitly specified on each operation.

It's not currently possible to get this information for BAR1 mappings,
so let's fix that ahead of time.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Will prevent spurious MMU fault interrupts if something decides to touch
BAR1 after we've unloaded the driver.

Exposed external to BAR so that INSTMEM can use it to better control the
suspend/resume fast-path access.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

If we want to be able to hit the instmem fast-path in a few trickier cases,
we need to be more flexible with when we can initialise BAR2 access.

There's probably a decent case to be made for merging BAR/INSTMEM into BUS,
but that's something to ponder another day.

Flushes have been added after the write to bind the instance block,
as later commits will reveal the need for them.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Will prevent spurious MMU fault interrupts if something decides to touch
BAR1 after we've unloaded the driver.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

BAR2 being done for practical reasons, this is just for consistency.

Flushes have been added after the write to bind the instance block,
as later commits will reveal the need for them.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

NVIDIA call it BAR2, Linux APIs treat it as BAR3 due to BAR1 being a
64-bit BAR, which I presume take two slots or something.

No actual code changes here, just to make future commits less messy.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

It's handled by FIFO preinit() now.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>