1. 29 Jun, 2010 18 commits
    • Tejun Heo's avatar
      workqueue: introduce global cwq and unify cwq locks · 8b03ae3c
      Tejun Heo authored
      There is one gcwq (global cwq) per each cpu and all cwqs on an cpu
      point to it.  A gcwq contains a lock to be used by all cwqs on the cpu
      and an ida to give IDs to workers belonging to the cpu.
      
      This patch introduces gcwq, moves worker_ida into gcwq and make all
      cwqs on the same cpu use the cpu's gcwq->lock instead of separate
      locks.  gcwq->ida is now protected by gcwq->lock too.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      8b03ae3c
    • Tejun Heo's avatar
      workqueue: reimplement workqueue freeze using max_active · a0a1a5fd
      Tejun Heo authored
      Currently, workqueue freezing is implemented by marking the worker
      freezeable and calling try_to_freeze() from dispatch loop.
      Reimplement it using cwq->limit so that the workqueue is frozen
      instead of the worker.
      
      * workqueue_struct->saved_max_active is added which stores the
        specified max_active on initialization.
      
      * On freeze, all cwq->max_active's are quenched to zero.  Freezing is
        complete when nr_active on all cwqs reach zero.
      
      * On thaw, all cwq->max_active's are restored to wq->saved_max_active
        and the worklist is repopulated.
      
      This new implementation allows having single shared pool of workers
      per cpu.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      a0a1a5fd
    • Tejun Heo's avatar
      workqueue: implement per-cwq active work limit · 1e19ffc6
      Tejun Heo authored
      Add cwq->nr_active, cwq->max_active and cwq->delayed_work.  nr_active
      counts the number of active works per cwq.  A work is active if it's
      flushable (colored) and is on cwq's worklist.  If nr_active reaches
      max_active, new works are queued on cwq->delayed_work and activated
      later as works on the cwq complete and decrement nr_active.
      
      cwq->max_active can be specified via the new @max_active parameter to
      __create_workqueue() and is set to 1 for all workqueues for now.  As
      each cwq has only single worker now, this double queueing doesn't
      cause any behavior difference visible to its users.
      
      This will be used to reimplement freeze/thaw and implement shared
      worker pool.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      1e19ffc6
    • Tejun Heo's avatar
      workqueue: reimplement work flushing using linked works · affee4b2
      Tejun Heo authored
      A work is linked to the next one by having WORK_STRUCT_LINKED bit set
      and these links can be chained.  When a linked work is dispatched to a
      worker, all linked works are dispatched to the worker's newly added
      ->scheduled queue and processed back-to-back.
      
      Currently, as there's only single worker per cwq, having linked works
      doesn't make any visible behavior difference.  This change is to
      prepare for multiple shared workers per cpu.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      affee4b2
    • Tejun Heo's avatar
      workqueue: introduce worker · c34056a3
      Tejun Heo authored
      Separate out worker thread related information to struct worker from
      struct cpu_workqueue_struct and implement helper functions to deal
      with the new struct worker.  The only change which is visible outside
      is that now workqueue worker are all named "kworker/CPUID:WORKERID"
      where WORKERID is allocated from per-cpu ida.
      
      This is in preparation of concurrency managed workqueue where shared
      multiple workers would be available per cpu.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      c34056a3
    • Tejun Heo's avatar
      workqueue: reimplement workqueue flushing using color coded works · 73f53c4a
      Tejun Heo authored
      Reimplement workqueue flushing using color coded works.  wq has the
      current work color which is painted on the works being issued via
      cwqs.  Flushing a workqueue is achieved by advancing the current work
      colors of cwqs and waiting for all the works which have any of the
      previous colors to drain.
      
      Currently there are 16 possible colors, one is reserved for no color
      and 15 colors are useable allowing 14 concurrent flushes.  When color
      space gets full, flush attempts are batched up and processed together
      when color frees up, so even with many concurrent flushers, the new
      implementation won't build up huge queue of flushers which has to be
      processed one after another.
      
      Only works which are queued via __queue_work() are colored.  Works
      which are directly put on queue using insert_work() use NO_COLOR and
      don't participate in workqueue flushing.  Currently only works used
      for work-specific flush fall in this category.
      
      This new implementation leaves only cleanup_workqueue_thread() as the
      user of flush_cpu_workqueue().  Just make its users use
      flush_workqueue() and kthread_stop() directly and kill
      cleanup_workqueue_thread().  As workqueue flushing doesn't use barrier
      request anymore, the comment describing the complex synchronization
      around it in cleanup_workqueue_thread() is removed together with the
      function.
      
      This new implementation is to allow having and sharing multiple
      workers per cpu.
      
      Please note that one more bit is reserved for a future work flag by
      this patch.  This is to avoid shifting bits and updating comments
      later.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      73f53c4a
    • Tejun Heo's avatar
      workqueue: update cwq alignement · 0f900049
      Tejun Heo authored
      work->data field is used for two purposes.  It points to cwq it's
      queued on and the lower bits are used for flags.  Currently, two bits
      are reserved which is always safe as 4 byte alignment is guaranteed on
      every architecture.  However, future changes will need more flag bits.
      
      On SMP, the percpu allocator is capable of honoring larger alignment
      (there are other users which depend on it) and larger alignment works
      just fine.  On UP, percpu allocator is a thin wrapper around
      kzalloc/kfree() and don't honor alignment request.
      
      This patch introduces WORK_STRUCT_FLAG_BITS and implements
      alloc/free_cwqs() which guarantees max(1 << WORK_STRUCT_FLAG_BITS,
      __alignof__(unsigned long long) alignment both on SMP and UP.  On SMP,
      simply wrapping percpu allocator is enough.  On UP, extra space is
      allocated so that cwq can be aligned and the original pointer can be
      stored after it which is used in the free path.
      
      * Alignment problem on UP is reported by Michal Simek.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Christoph Lameter <cl@linux-foundation.org>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Reported-by: default avatarMichal Simek <michal.simek@petalogix.com>
      0f900049
    • Tejun Heo's avatar
      workqueue: kill cpu_populated_map · 1537663f
      Tejun Heo authored
      Worker management is about to be overhauled.  Simplify things by
      removing cpu_populated_map, creating workers for all possible cpus and
      making single threaded workqueues behave more like multi threaded
      ones.
      
      After this patch, all cwqs are always initialized, all workqueues are
      linked on the workqueues list and workers for all possibles cpus
      always exist.  This also makes CPU hotplug support simpler - checking
      ->cpus_allowed before processing works in worker_thread() and flushing
      cwqs on CPU_POST_DEAD are enough.
      
      While at it, make get_cwq() always return the cwq for the specified
      cpu, add target_cwq() for cases where single thread distinction is
      necessary and drop all direct usage of per_cpu_ptr() on wq->cpu_wq.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      1537663f
    • Tejun Heo's avatar
      workqueue: temporarily remove workqueue tracing · 64166699
      Tejun Heo authored
      Strip tracing code from workqueue and remove workqueue tracing.  This
      is temporary measure till concurrency managed workqueue is complete.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      64166699
    • Tejun Heo's avatar
      workqueue: separate out process_one_work() · a62428c0
      Tejun Heo authored
      Separate out process_one_work() out of run_workqueue().  This patch
      doesn't cause any behavior change.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      a62428c0
    • Tejun Heo's avatar
      workqueue: define masks for work flags and conditionalize STATIC flags · 22df02bb
      Tejun Heo authored
      Work flags are about to see more traditional mask handling.  Define
      WORK_STRUCT_*_BIT as the bit position constant and redefine
      WORK_STRUCT_* as bit masks.  Also, make WORK_STRUCT_STATIC_* flags
      conditional
      
      While at it, re-define these constants as enums and use
      WORK_STRUCT_STATIC instead of hard-coding 2 in
      WORK_DATA_STATIC_INIT().
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      22df02bb
    • Tejun Heo's avatar
      workqueue: merge feature parameters into flags · 97e37d7b
      Tejun Heo authored
      Currently, __create_workqueue_key() takes @singlethread and
      @freezeable paramters and store them separately in workqueue_struct.
      Merge them into a single flags parameter and field and use
      WQ_FREEZEABLE and WQ_SINGLE_THREAD.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      97e37d7b
    • Tejun Heo's avatar
      workqueue: misc/cosmetic updates · 4690c4ab
      Tejun Heo authored
      Make the following updates in preparation of concurrency managed
      workqueue.  None of these changes causes any visible behavior
      difference.
      
      * Add comments and adjust indentations to data structures and several
        functions.
      
      * Rename wq_per_cpu() to get_cwq() and swap the position of two
        parameters for consistency.  Convert a direct per_cpu_ptr() access
        to wq->cpu_wq to get_cwq().
      
      * Add work_static() and Update set_wq_data() such that it sets the
        flags part to WORK_STRUCT_PENDING | WORK_STRUCT_STATIC if static |
        @extra_flags.
      
      * Move santiy check on work->entry emptiness from queue_work_on() to
        __queue_work() which all queueing paths share.
      
      * Make __queue_work() take @cpu and @wq instead of @cwq.
      
      * Restructure flush_work() and __create_workqueue_key() to make them
        easier to modify.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      4690c4ab
    • Tejun Heo's avatar
      workqueue: kill RT workqueue · c790bce0
      Tejun Heo authored
      With stop_machine() converted to use cpu_stop, RT workqueue doesn't
      have any user left.  Kill RT workqueue support.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      c790bce0
    • Tejun Heo's avatar
      acpi: use queue_work_on() instead of binding workqueue worker to cpu0 · 8fec62b2
      Tejun Heo authored
      ACPI works need to be executed on cpu0 and acpi/osl.c achieves this by
      creating singlethread workqueue and then binding it to cpu0 from a
      work which is quite unorthodox.  Make it create regular workqueues and
      use queue_work_on() instead.  This is in preparation of concurrency
      managed workqueue and the extra workers won't be a problem after it's
      implemented.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      8fec62b2
    • Tejun Heo's avatar
      kthread: implement kthread_data() · 82805ab7
      Tejun Heo authored
      Implement kthread_data() which takes @task pointing to a kthread and
      returns @data specified when creating the kthread.  The caller is
      responsible for ensuring the validity of @task when calling this
      function.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      82805ab7
    • Tejun Heo's avatar
      ivtv: use kthread_worker instead of workqueue · 7bc46560
      Tejun Heo authored
      Upcoming workqueue updates will no longer guarantee fixed workqueue to
      worker kthread association, so giving RT priority to the irq worker
      won't work.  Use kthread_worker which guarantees specific kthread
      association instead.  This also makes setting the priority cleaner.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Andy Walls <awalls@md.metrocast.net>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: ivtv-devel@ivtvdriver.org
      Cc: linux-media@vger.kernel.org
      7bc46560
    • Tejun Heo's avatar
      kthread: implement kthread_worker · b56c0d89
      Tejun Heo authored
      Implement simple work processor for kthread.  This is to ease using
      kthread.  Single thread workqueue used to be used for things like this
      but workqueue won't guarantee fixed kthread association anymore to
      enable worker sharing.
      
      This can be used in cases where specific kthread association is
      necessary, for example, when it should have RT priority or be assigned
      to certain cgroup.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      b56c0d89
  2. 13 Jun, 2010 1 commit
  3. 12 Jun, 2010 1 commit
  4. 11 Jun, 2010 20 commits