1. 14 Jul, 2010 1 commit
  2. 02 Jul, 2010 8 commits
    • Tejun Heo's avatar
      workqueue: remove WQ_SINGLE_CPU and use WQ_UNBOUND instead · c7fc77f7
      Tejun Heo authored
      WQ_SINGLE_CPU combined with @max_active of 1 is used to achieve full
      ordering among works queued to a workqueue.  The same can be achieved
      using WQ_UNBOUND as unbound workqueues always use the gcwq for
      WORK_CPU_UNBOUND.  As @max_active is always one and benefits from cpu
      locality isn't accessible anyway, serving them with unbound workqueues
      should be fine.
      
      Drop WQ_SINGLE_CPU support and use WQ_UNBOUND instead.  Note that most
      single thread workqueue users will be converted to use multithread or
      non-reentrant instead and only the ones which require strict ordering
      will keep using WQ_UNBOUND + @max_active of 1.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      c7fc77f7
    • Tejun Heo's avatar
      workqueue: implement unbound workqueue · f3421797
      Tejun Heo authored
      This patch implements unbound workqueue which can be specified with
      WQ_UNBOUND flag on creation.  An unbound workqueue has the following
      properties.
      
      * It uses a dedicated gcwq with a pseudo CPU number WORK_CPU_UNBOUND.
        This gcwq is always online and disassociated.
      
      * Workers are not bound to any CPU and not concurrency managed.  Works
        are dispatched to workers as soon as possible and the only applied
        limitation is @max_active.  IOW, all unbound workqeueues are
        implicitly high priority.
      
      Unbound workqueues can be used as simple execution context provider.
      Contexts unbound to any cpu are served as soon as possible.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Arjan van de Ven <arjan@linux.intel.com>
      Cc: David Howells <dhowells@redhat.com>
      f3421797
    • Tejun Heo's avatar
      workqueue: prepare for WQ_UNBOUND implementation · bdbc5dd7
      Tejun Heo authored
      In preparation of WQ_UNBOUND addition, make the following changes.
      
      * Add WORK_CPU_* constants for pseudo cpu id numbers used (currently
        only WORK_CPU_NONE) and use them instead of NR_CPUS.  This is to
        allow another pseudo cpu id for unbound cpu.
      
      * Reorder WQ_* flags.
      
      * Make workqueue_struct->cpu_wq a union which contains a percpu
        pointer, regular pointer and an unsigned long value and use
        kzalloc/kfree() in UP allocation path.  This will be used to
        implement unbound workqueues which will use only one cwq on SMPs.
      
      * Move alloc_cwqs() allocation after initialization of wq fields, so
        that alloc_cwqs() has access to wq->flags.
      
      * Trivial relocation of wq local variables in freeze functions.
      
      These changes don't cause any functional change.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      bdbc5dd7
    • Tejun Heo's avatar
      libata: take advantage of cmwq and remove concurrency limitations · ad72cf98
      Tejun Heo authored
      libata has two concurrency related limitations.
      
      a. ata_wq which is used for polling PIO has single thread per CPU.  If
         there are multiple devices doing polling PIO on the same CPU, they
         can't be executed simultaneously.
      
      b. ata_aux_wq which is used for SCSI probing has single thread.  In
         cases where SCSI probing is stalled for extended period of time
         which is possible for ATAPI devices, this will stall all probing.
      
      #a is solved by increasing maximum concurrency of ata_wq.  Please note
      that polling PIO might be used under allocation path and thus needs to
      be served by a separate wq with a rescuer.
      
      #b is solved by using the default wq instead and achieving exclusion
      via per-port mutex.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Acked-by: default avatarJeff Garzik <jgarzik@pobox.com>
      ad72cf98
    • Tejun Heo's avatar
      workqueue: fix worker management invocation without pending works · d313dd85
      Tejun Heo authored
      When there's no pending work to do, worker_thread() goes back to sleep
      after waking up without checking whether worker management is
      necessary.  This means that idle worker exit requests can be ignored
      if the gcwq stays empty.
      
      Fix it by making worker_thread() always check whether worker
      management is necessary before going to sleep.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      d313dd85
    • Tejun Heo's avatar
      workqueue: fix incorrect cpu number BUG_ON() in get_work_gcwq() · a1e453d2
      Tejun Heo authored
      get_work_gcwq() was incorrectly triggering BUG_ON() if cpu number is
      equal to or higher than num_possible_cpus() instead of nr_cpu_ids.
      Fix it.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      a1e453d2
    • Tejun Heo's avatar
      workqueue: fix race condition in flush_workqueue() · 4ce48b37
      Tejun Heo authored
      When one flusher is cascading to the next flusher, it first sets
      wq->first_flusher to the next one and sets up the next flush cycle.
      If there's nothing to do for the next cycle, it clears
      wq->flush_flusher and proceeds to the one after that.
      
      If the woken up flusher checks wq->first_flusher before it gets
      cleared, it will incorrectly assume the role of the first flusher,
      which triggers BUG_ON() sanity check.
      
      Fix it by checking wq->first_flusher again after grabbing the mutex.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      4ce48b37
    • Tejun Heo's avatar
      workqueue: use worker_set/clr_flags() only from worker itself · cb444766
      Tejun Heo authored
      worker_set/clr_flags() assume that if none of NOT_RUNNING flags is set
      the worker must be contributing to nr_running which is only true if
      the worker is actually running.
      
      As when called from self, it is guaranteed that the worker is running,
      those functions can be safely used from the worker itself and they
      aren't necessary from other places anyway.  Make the following changes
      to fix the bug.
      
      * Make worker_set/clr_flags() whine if not called from self.
      
      * Convert all places which called those functions from other tasks to
        manipulate flags directly.
      
      * Make trustee_thread() directly clear nr_running after setting
        WORKER_ROGUE on all workers.  This is the only place where
        nr_running manipulation is necessary outside of workers themselves.
      
      * While at it, add sanity check for nr_running in worker_enter_idle().
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      cb444766
  3. 29 Jun, 2010 31 commits
    • Tejun Heo's avatar
      workqueue: implement cpu intensive workqueue · fb0e7beb
      Tejun Heo authored
      This patch implements cpu intensive workqueue which can be specified
      with WQ_CPU_INTENSIVE flag on creation.  Works queued to a cpu
      intensive workqueue don't participate in concurrency management.  IOW,
      it doesn't contribute to gcwq->nr_running and thus doesn't delay
      excution of other works.
      
      Note that although cpu intensive works won't delay other works, they
      can be delayed by other works.  Combine with WQ_HIGHPRI to avoid being
      delayed by other works too.
      
      As the name suggests this is useful when using workqueue for cpu
      intensive works.  Workers executing cpu intensive works are not
      considered for workqueue concurrency management and left for the
      scheduler to manage.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      fb0e7beb
    • Tejun Heo's avatar
      workqueue: implement high priority workqueue · 649027d7
      Tejun Heo authored
      This patch implements high priority workqueue which can be specified
      with WQ_HIGHPRI flag on creation.  A high priority workqueue has the
      following properties.
      
      * A work queued to it is queued at the head of the worklist of the
        respective gcwq after other highpri works, while normal works are
        always appended at the end.
      
      * As long as there are highpri works on gcwq->worklist,
        [__]need_more_worker() remains %true and process_one_work() wakes up
        another worker before it start executing a work.
      
      The above two properties guarantee that works queued to high priority
      workqueues are dispatched to workers and start execution as soon as
      possible regardless of the state of other works.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Andi Kleen <andi@firstfloor.org>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      649027d7
    • Tejun Heo's avatar
      workqueue: implement several utility APIs · dcd989cb
      Tejun Heo authored
      Implement the following utility APIs.
      
       workqueue_set_max_active()	: adjust max_active of a wq
       workqueue_congested()		: test whether a wq is contested
       work_cpu()			: determine the last / current cpu of a work
       work_busy()			: query whether a work is busy
      
      * Anton Blanchard fixed missing ret initialization in work_busy().
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Anton Blanchard <anton@samba.org>
      dcd989cb
    • Tejun Heo's avatar
      workqueue: s/__create_workqueue()/alloc_workqueue()/, and add system workqueues · d320c038
      Tejun Heo authored
      This patch makes changes to make new workqueue features available to
      its users.
      
      * Now that workqueue is more featureful, there should be a public
        workqueue creation function which takes paramters to control them.
        Rename __create_workqueue() to alloc_workqueue() and make 0
        max_active mean WQ_DFL_ACTIVE.  In the long run, all
        create_workqueue_*() will be converted over to alloc_workqueue().
      
      * To further unify access interface, rename keventd_wq to system_wq
        and export it.
      
      * Add system_long_wq and system_nrt_wq.  The former is to host long
        running works separately (so that flush_scheduled_work() dosen't
        take so long) and the latter guarantees any queued work item is
        never executed in parallel by multiple CPUs.  These will be used by
        future patches to update workqueue users.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      d320c038
    • Tejun Heo's avatar
      workqueue: increase max_active of keventd and kill current_is_keventd() · b71ab8c2
      Tejun Heo authored
      Define WQ_MAX_ACTIVE and create keventd with max_active set to half of
      it which means that keventd now can process upto WQ_MAX_ACTIVE / 2 - 1
      works concurrently.  Unless some combination can result in dependency
      loop longer than max_active, deadlock won't happen and thus it's
      unnecessary to check whether current_is_keventd() before trying to
      schedule a work.  Kill current_is_keventd().
      
      (Lockdep annotations are broken.  We need lock_map_acquire_read_norecurse())
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: Christoph Lameter <cl@linux-foundation.org>
      Cc: Tony Luck <tony.luck@intel.com>
      Cc: Andi Kleen <ak@linux.intel.com>
      Cc: Oleg Nesterov <oleg@redhat.com>
      b71ab8c2
    • Tejun Heo's avatar
      workqueue: implement concurrency managed dynamic worker pool · e22bee78
      Tejun Heo authored
      Instead of creating a worker for each cwq and putting it into the
      shared pool, manage per-cpu workers dynamically.
      
      Works aren't supposed to be cpu cycle hogs and maintaining just enough
      concurrency to prevent work processing from stalling due to lack of
      processing context is optimal.  gcwq keeps the number of concurrent
      active workers to minimum but no less.  As long as there's one or more
      running workers on the cpu, no new worker is scheduled so that works
      can be processed in batch as much as possible but when the last
      running worker blocks, gcwq immediately schedules new worker so that
      the cpu doesn't sit idle while there are works to be processed.
      
      gcwq always keeps at least single idle worker around.  When a new
      worker is necessary and the worker is the last idle one, the worker
      assumes the role of "manager" and manages the worker pool -
      ie. creates another worker.  Forward-progress is guaranteed by having
      dedicated rescue workers for workqueues which may be necessary while
      creating a new worker.  When the manager is having problem creating a
      new worker, mayday timer activates and rescue workers are summoned to
      the cpu and execute works which might be necessary to create new
      workers.
      
      Trustee is expanded to serve the role of manager while a CPU is being
      taken down and stays down.  As no new works are supposed to be queued
      on a dead cpu, it just needs to drain all the existing ones.  Trustee
      continues to try to create new workers and summon rescuers as long as
      there are pending works.  If the CPU is brought back up while the
      trustee is still trying to drain the gcwq from the previous offlining,
      the trustee will kill all idles ones and tell workers which are still
      busy to rebind to the cpu, and pass control over to gcwq which assumes
      the manager role as necessary.
      
      Concurrency managed worker pool reduces the number of workers
      drastically.  Only workers which are necessary to keep the processing
      going are created and kept.  Also, it reduces cache footprint by
      avoiding unnecessarily switching contexts between different workers.
      
      Please note that this patch does not increase max_active of any
      workqueue.  All workqueues can still only process one work per cpu.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      e22bee78
    • Tejun Heo's avatar
      workqueue: implement worker_{set|clr}_flags() · d302f017
      Tejun Heo authored
      Implement worker_{set|clr}_flags() to manipulate worker flags.  These
      are currently simple wrappers but logics to track the current worker
      state and the current level of concurrency will be added.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      d302f017
    • Tejun Heo's avatar
      workqueue: use shared worklist and pool all workers per cpu · 7e11629d
      Tejun Heo authored
      Use gcwq->worklist instead of cwq->worklist and break the strict
      association between a cwq and its worker.  All works queued on a cpu
      are queued on gcwq->worklist and processed by any available worker on
      the gcwq.
      
      As there no longer is strict association between a cwq and its worker,
      whether a work is executing can now only be determined by calling
      [__]find_worker_executing_work().
      
      After this change, the only association between a cwq and its worker
      is that a cwq puts a worker into shared worker pool on creation and
      kills it on destruction.  As all workqueues are still limited to
      max_active of one, this means that there are always at least as many
      workers as active works and thus there's no danger for deadlock.
      
      The break of strong association between cwqs and workers requires
      somewhat clumsy changes to current_is_keventd() and
      destroy_workqueue().  Dynamic worker pool management will remove both
      clumsy changes.  current_is_keventd() won't be necessary at all as the
      only reason it exists is to avoid queueing a work from a work which
      will be allowed just fine.  The clumsy part of destroy_workqueue() is
      added because a worker can only be destroyed while idle and there's no
      guarantee a worker is idle when its wq is going down.  With dynamic
      pool management, workers are not associated with workqueues at all and
      only idle ones will be submitted to destroy_workqueue() so the code
      won't be necessary anymore.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      7e11629d
    • Tejun Heo's avatar
      workqueue: implement WQ_NON_REENTRANT · 18aa9eff
      Tejun Heo authored
      With gcwq managing all the workers and work->data pointing to the last
      gcwq it was on, non-reentrance can be easily implemented by checking
      whether the work is still running on the previous gcwq on queueing.
      Implement it.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      18aa9eff
    • Tejun Heo's avatar
      workqueue: carry cpu number in work data once execution starts · 7a22ad75
      Tejun Heo authored
      To implement non-reentrant workqueue, the last gcwq a work was
      executed on must be reliably obtainable as long as the work structure
      is valid even if the previous workqueue has been destroyed.
      
      To achieve this, work->data will be overloaded to carry the last cpu
      number once execution starts so that the previous gcwq can be located
      reliably.  This means that cwq can't be obtained from work after
      execution starts but only gcwq.
      
      Implement set_work_{cwq|cpu}(), get_work_[g]cwq() and
      clear_work_data() to set work data to the cpu number when starting
      execution, access the overloaded work data and clear it after
      cancellation.
      
      queue_delayed_work_on() is updated to preserve the last cpu while
      in-flight in timer and other callers which depended on getting cwq
      from work after execution starts are converted to depend on gcwq
      instead.
      
      * Anton Blanchard fixed compile error on powerpc due to missing
        linux/threads.h include.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Anton Blanchard <anton@samba.org>
      7a22ad75
    • Tejun Heo's avatar
      workqueue: add find_worker_executing_work() and track current_cwq · 8cca0eea
      Tejun Heo authored
      Now that all the workers are tracked by gcwq, we can find which worker
      is executing a work from gcwq.  Implement find_worker_executing_work()
      and make worker track its current_cwq so that we can find things the
      other way around.  This will be used to implement non-reentrant wqs.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      8cca0eea
    • Tejun Heo's avatar
      workqueue: make single thread workqueue shared worker pool friendly · 502ca9d8
      Tejun Heo authored
      Reimplement st (single thread) workqueue so that it's friendly to
      shared worker pool.  It was originally implemented by confining st
      workqueues to use cwq of a fixed cpu and always having a worker for
      the cpu.  This implementation isn't very friendly to shared worker
      pool and suboptimal in that it ends up crossing cpu boundaries often.
      
      Reimplement st workqueue using dynamic single cpu binding and
      cwq->limit.  WQ_SINGLE_THREAD is replaced with WQ_SINGLE_CPU.  In a
      single cpu workqueue, at most single cwq is bound to the wq at any
      given time.  Arbitration is done using atomic accesses to
      wq->single_cpu when queueing a work.  Once bound, the binding stays
      till the workqueue is drained.
      
      Note that the binding is never broken while a workqueue is frozen.
      This is because idle cwqs may have works waiting in delayed_works
      queue while frozen.  On thaw, the cwq is restarted if there are any
      delayed works or unbound otherwise.
      
      When combined with max_active limit of 1, single cpu workqueue has
      exactly the same execution properties as the original single thread
      workqueue while allowing sharing of per-cpu workers.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      502ca9d8
    • Tejun Heo's avatar
      workqueue: reimplement CPU hotplugging support using trustee · db7bccf4
      Tejun Heo authored
      Reimplement CPU hotplugging support using trustee thread.  On CPU
      down, a trustee thread is created and each step of CPU down is
      executed by the trustee and workqueue_cpu_callback() simply drives and
      waits for trustee state transitions.
      
      CPU down operation no longer waits for works to be drained but trustee
      sticks around till all pending works have been completed.  If CPU is
      brought back up while works are still draining,
      workqueue_cpu_callback() tells trustee to step down and tell workers
      to rebind to the cpu.
      
      As it's difficult to tell whether cwqs are empty if it's freezing or
      frozen, trustee doesn't consider draining to be complete while a gcwq
      is freezing or frozen (tracked by new GCWQ_FREEZING flag).  Also,
      workers which get unbound from their cpu are marked with WORKER_ROGUE.
      
      Trustee based implementation doesn't bring any new feature at this
      point but it will be used to manage worker pool when dynamic shared
      worker pool is implemented.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      db7bccf4
    • Tejun Heo's avatar
      workqueue: implement worker states · c8e55f36
      Tejun Heo authored
      Implement worker states.  After created, a worker is STARTED.  While a
      worker isn't processing a work, it's IDLE and chained on
      gcwq->idle_list.  While processing a work, a worker is BUSY and
      chained on gcwq->busy_hash.  Also, gcwq now counts the number of all
      workers and idle ones.
      
      worker_thread() is restructured to reflect state transitions.
      cwq->more_work is removed and waking up a worker makes it check for
      events.  A worker is killed by setting DIE flag while it's IDLE and
      waking it up.
      
      This gives gcwq better visibility of what's going on and allows it to
      find out whether a work is executing quickly which is necessary to
      have multiple workers processing the same cwq.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      c8e55f36
    • Tejun Heo's avatar
      workqueue: introduce global cwq and unify cwq locks · 8b03ae3c
      Tejun Heo authored
      There is one gcwq (global cwq) per each cpu and all cwqs on an cpu
      point to it.  A gcwq contains a lock to be used by all cwqs on the cpu
      and an ida to give IDs to workers belonging to the cpu.
      
      This patch introduces gcwq, moves worker_ida into gcwq and make all
      cwqs on the same cpu use the cpu's gcwq->lock instead of separate
      locks.  gcwq->ida is now protected by gcwq->lock too.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      8b03ae3c
    • Tejun Heo's avatar
      workqueue: reimplement workqueue freeze using max_active · a0a1a5fd
      Tejun Heo authored
      Currently, workqueue freezing is implemented by marking the worker
      freezeable and calling try_to_freeze() from dispatch loop.
      Reimplement it using cwq->limit so that the workqueue is frozen
      instead of the worker.
      
      * workqueue_struct->saved_max_active is added which stores the
        specified max_active on initialization.
      
      * On freeze, all cwq->max_active's are quenched to zero.  Freezing is
        complete when nr_active on all cwqs reach zero.
      
      * On thaw, all cwq->max_active's are restored to wq->saved_max_active
        and the worklist is repopulated.
      
      This new implementation allows having single shared pool of workers
      per cpu.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      a0a1a5fd
    • Tejun Heo's avatar
      workqueue: implement per-cwq active work limit · 1e19ffc6
      Tejun Heo authored
      Add cwq->nr_active, cwq->max_active and cwq->delayed_work.  nr_active
      counts the number of active works per cwq.  A work is active if it's
      flushable (colored) and is on cwq's worklist.  If nr_active reaches
      max_active, new works are queued on cwq->delayed_work and activated
      later as works on the cwq complete and decrement nr_active.
      
      cwq->max_active can be specified via the new @max_active parameter to
      __create_workqueue() and is set to 1 for all workqueues for now.  As
      each cwq has only single worker now, this double queueing doesn't
      cause any behavior difference visible to its users.
      
      This will be used to reimplement freeze/thaw and implement shared
      worker pool.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      1e19ffc6
    • Tejun Heo's avatar
      workqueue: reimplement work flushing using linked works · affee4b2
      Tejun Heo authored
      A work is linked to the next one by having WORK_STRUCT_LINKED bit set
      and these links can be chained.  When a linked work is dispatched to a
      worker, all linked works are dispatched to the worker's newly added
      ->scheduled queue and processed back-to-back.
      
      Currently, as there's only single worker per cwq, having linked works
      doesn't make any visible behavior difference.  This change is to
      prepare for multiple shared workers per cpu.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      affee4b2
    • Tejun Heo's avatar
      workqueue: introduce worker · c34056a3
      Tejun Heo authored
      Separate out worker thread related information to struct worker from
      struct cpu_workqueue_struct and implement helper functions to deal
      with the new struct worker.  The only change which is visible outside
      is that now workqueue worker are all named "kworker/CPUID:WORKERID"
      where WORKERID is allocated from per-cpu ida.
      
      This is in preparation of concurrency managed workqueue where shared
      multiple workers would be available per cpu.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      c34056a3
    • Tejun Heo's avatar
      workqueue: reimplement workqueue flushing using color coded works · 73f53c4a
      Tejun Heo authored
      Reimplement workqueue flushing using color coded works.  wq has the
      current work color which is painted on the works being issued via
      cwqs.  Flushing a workqueue is achieved by advancing the current work
      colors of cwqs and waiting for all the works which have any of the
      previous colors to drain.
      
      Currently there are 16 possible colors, one is reserved for no color
      and 15 colors are useable allowing 14 concurrent flushes.  When color
      space gets full, flush attempts are batched up and processed together
      when color frees up, so even with many concurrent flushers, the new
      implementation won't build up huge queue of flushers which has to be
      processed one after another.
      
      Only works which are queued via __queue_work() are colored.  Works
      which are directly put on queue using insert_work() use NO_COLOR and
      don't participate in workqueue flushing.  Currently only works used
      for work-specific flush fall in this category.
      
      This new implementation leaves only cleanup_workqueue_thread() as the
      user of flush_cpu_workqueue().  Just make its users use
      flush_workqueue() and kthread_stop() directly and kill
      cleanup_workqueue_thread().  As workqueue flushing doesn't use barrier
      request anymore, the comment describing the complex synchronization
      around it in cleanup_workqueue_thread() is removed together with the
      function.
      
      This new implementation is to allow having and sharing multiple
      workers per cpu.
      
      Please note that one more bit is reserved for a future work flag by
      this patch.  This is to avoid shifting bits and updating comments
      later.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      73f53c4a
    • Tejun Heo's avatar
      workqueue: update cwq alignement · 0f900049
      Tejun Heo authored
      work->data field is used for two purposes.  It points to cwq it's
      queued on and the lower bits are used for flags.  Currently, two bits
      are reserved which is always safe as 4 byte alignment is guaranteed on
      every architecture.  However, future changes will need more flag bits.
      
      On SMP, the percpu allocator is capable of honoring larger alignment
      (there are other users which depend on it) and larger alignment works
      just fine.  On UP, percpu allocator is a thin wrapper around
      kzalloc/kfree() and don't honor alignment request.
      
      This patch introduces WORK_STRUCT_FLAG_BITS and implements
      alloc/free_cwqs() which guarantees max(1 << WORK_STRUCT_FLAG_BITS,
      __alignof__(unsigned long long) alignment both on SMP and UP.  On SMP,
      simply wrapping percpu allocator is enough.  On UP, extra space is
      allocated so that cwq can be aligned and the original pointer can be
      stored after it which is used in the free path.
      
      * Alignment problem on UP is reported by Michal Simek.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Christoph Lameter <cl@linux-foundation.org>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Reported-by: default avatarMichal Simek <michal.simek@petalogix.com>
      0f900049
    • Tejun Heo's avatar
      workqueue: kill cpu_populated_map · 1537663f
      Tejun Heo authored
      Worker management is about to be overhauled.  Simplify things by
      removing cpu_populated_map, creating workers for all possible cpus and
      making single threaded workqueues behave more like multi threaded
      ones.
      
      After this patch, all cwqs are always initialized, all workqueues are
      linked on the workqueues list and workers for all possibles cpus
      always exist.  This also makes CPU hotplug support simpler - checking
      ->cpus_allowed before processing works in worker_thread() and flushing
      cwqs on CPU_POST_DEAD are enough.
      
      While at it, make get_cwq() always return the cwq for the specified
      cpu, add target_cwq() for cases where single thread distinction is
      necessary and drop all direct usage of per_cpu_ptr() on wq->cpu_wq.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      1537663f
    • Tejun Heo's avatar
      workqueue: temporarily remove workqueue tracing · 64166699
      Tejun Heo authored
      Strip tracing code from workqueue and remove workqueue tracing.  This
      is temporary measure till concurrency managed workqueue is complete.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      64166699
    • Tejun Heo's avatar
      workqueue: separate out process_one_work() · a62428c0
      Tejun Heo authored
      Separate out process_one_work() out of run_workqueue().  This patch
      doesn't cause any behavior change.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      a62428c0
    • Tejun Heo's avatar
      workqueue: define masks for work flags and conditionalize STATIC flags · 22df02bb
      Tejun Heo authored
      Work flags are about to see more traditional mask handling.  Define
      WORK_STRUCT_*_BIT as the bit position constant and redefine
      WORK_STRUCT_* as bit masks.  Also, make WORK_STRUCT_STATIC_* flags
      conditional
      
      While at it, re-define these constants as enums and use
      WORK_STRUCT_STATIC instead of hard-coding 2 in
      WORK_DATA_STATIC_INIT().
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      22df02bb
    • Tejun Heo's avatar
      workqueue: merge feature parameters into flags · 97e37d7b
      Tejun Heo authored
      Currently, __create_workqueue_key() takes @singlethread and
      @freezeable paramters and store them separately in workqueue_struct.
      Merge them into a single flags parameter and field and use
      WQ_FREEZEABLE and WQ_SINGLE_THREAD.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      97e37d7b
    • Tejun Heo's avatar
      workqueue: misc/cosmetic updates · 4690c4ab
      Tejun Heo authored
      Make the following updates in preparation of concurrency managed
      workqueue.  None of these changes causes any visible behavior
      difference.
      
      * Add comments and adjust indentations to data structures and several
        functions.
      
      * Rename wq_per_cpu() to get_cwq() and swap the position of two
        parameters for consistency.  Convert a direct per_cpu_ptr() access
        to wq->cpu_wq to get_cwq().
      
      * Add work_static() and Update set_wq_data() such that it sets the
        flags part to WORK_STRUCT_PENDING | WORK_STRUCT_STATIC if static |
        @extra_flags.
      
      * Move santiy check on work->entry emptiness from queue_work_on() to
        __queue_work() which all queueing paths share.
      
      * Make __queue_work() take @cpu and @wq instead of @cwq.
      
      * Restructure flush_work() and __create_workqueue_key() to make them
        easier to modify.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      4690c4ab
    • Tejun Heo's avatar
      workqueue: kill RT workqueue · c790bce0
      Tejun Heo authored
      With stop_machine() converted to use cpu_stop, RT workqueue doesn't
      have any user left.  Kill RT workqueue support.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      c790bce0
    • Tejun Heo's avatar
      acpi: use queue_work_on() instead of binding workqueue worker to cpu0 · 8fec62b2
      Tejun Heo authored
      ACPI works need to be executed on cpu0 and acpi/osl.c achieves this by
      creating singlethread workqueue and then binding it to cpu0 from a
      work which is quite unorthodox.  Make it create regular workqueues and
      use queue_work_on() instead.  This is in preparation of concurrency
      managed workqueue and the extra workers won't be a problem after it's
      implemented.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      8fec62b2
    • Tejun Heo's avatar
      kthread: implement kthread_data() · 82805ab7
      Tejun Heo authored
      Implement kthread_data() which takes @task pointing to a kthread and
      returns @data specified when creating the kthread.  The caller is
      responsible for ensuring the validity of @task when calling this
      function.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      82805ab7
    • Tejun Heo's avatar
      ivtv: use kthread_worker instead of workqueue · 7bc46560
      Tejun Heo authored
      Upcoming workqueue updates will no longer guarantee fixed workqueue to
      worker kthread association, so giving RT priority to the irq worker
      won't work.  Use kthread_worker which guarantees specific kthread
      association instead.  This also makes setting the priority cleaner.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Andy Walls <awalls@md.metrocast.net>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: ivtv-devel@ivtvdriver.org
      Cc: linux-media@vger.kernel.org
      7bc46560