Commit ce75e16e authored by Kirill Smelkov's avatar Kirill Smelkov Committed by Romain Courteaud

component/python-2.7: Lib/subprocess: Speedup close_fds=True

@romain reports that Popen(close_fds=True) is slow on py2. Let's semantically
backport from py3 how to close only actually opened file descriptors instead of
whole 3..`ulimit -n` range.

Attached test benchmark shows the following results with `ulimit -n`=65K:

Before this patch:

    $ ./bin/python2.7 ~/x.py
    close_fds=False:
    0.001251 s/call
    0.001337 s/call
    0.001486 s/call

    close_fds=True:
    0.017973 s/call
    0.018152 s/call
    0.018204 s/call

After the patch:

    $ ./bin/python2.7 ~/x.py
    close_fds=False:
    0.001391 s/call
    0.001416 s/call
    0.001570 s/call

    close_fds=True:
    0.001469 s/call
    0.001479 s/call
    0.001491 s/call

i.e. ~12x speedup.

References on this subject are in the patch itself.

The test benchmark is below:

---- 8< ----
import timeit
from subprocess import check_call

def f():
    check_call(['true'], close_fds=False)
def g():
    check_call(['true'], close_fds=True)

N=3
n=100
print 'close_fds=False:'
for i in range(N):
    print '%.6f s/call' % (timeit.timeit(f, number=n) / n)
print
print 'close_fds=True:'
for i in range(N):
    print '%.6f s/call' % (timeit.timeit(g, number=n) / n)

/helped-by @jm
parent 7cf8d439
Pipeline #19119 failed with stage
in 0 seconds
...@@ -42,6 +42,7 @@ patches = ...@@ -42,6 +42,7 @@ patches =
${:_profile_base_location_}/pytracemalloc_pep445.patch#9f3145817afa2b7fad801fde8447e396 ${:_profile_base_location_}/pytracemalloc_pep445.patch#9f3145817afa2b7fad801fde8447e396
${:_profile_base_location_}/disabled_module_list.patch#e038a8016475574c810cbaaf0e42f4ac ${:_profile_base_location_}/disabled_module_list.patch#e038a8016475574c810cbaaf0e42f4ac
${:_profile_base_location_}/asyncore_poll_insteadof_select.patch#ab6991c0ee6e25aeb8951e71f280a2f1 ${:_profile_base_location_}/asyncore_poll_insteadof_select.patch#ab6991c0ee6e25aeb8951e71f280a2f1
${:_profile_base_location_}/py27-subproc-closefds-fast.patch#e495e44491694a8972da11739206f2e6
url = url =
http://www.python.org/ftp/python/${:package_version}/Python-${:package_version}${:package_version_suffix}.tar.xz http://www.python.org/ftp/python/${:package_version}/Python-${:package_version}${:package_version_suffix}.tar.xz
configure-options = configure-options =
......
From 9a79fa64c6298998d04a5ae6d699222a93cd0cb8 Mon Sep 17 00:00:00 2001
From: Kirill Smelkov <kirr@nexedi.com>
Date: Fri, 24 Dec 2021 13:55:08 +0300
Subject: [PATCH] [py27] subprocess: Use /proc/self/fd/ to know opened file
descriptors
... to avoid O(MAXFD) slowdown on closefds=True.
This is semantic backport of what Python3 does.
---
Lib/subprocess.py | 23 ++++++++++++++++++++++-
1 file changed, 22 insertions(+), 1 deletion(-)
diff --git a/Lib/subprocess.py b/Lib/subprocess.py
index 1f2da0ffbe8..300557220ac 100644
--- a/Lib/subprocess.py
+++ b/Lib/subprocess.py
@@ -869,7 +869,28 @@ def pipe_cloexec(self):
def _close_fds(self, but):
- if hasattr(os, 'closerange'):
+ # try to retrieve list of opened file descriptors to avoid being slow on large MAXFD
+ # https://www.erp5.com/project_section/vifib/forum/python2--subprocess.Popen-is-slow-when-close_fds-True-and-nofile-ulimit-is-high-PO01sREVXn
+ # https://bugs.python.org/issue8052
+ # https://github.com/python/cpython/commit/8facece99a59
+ fdv = None
+ try:
+ fdv = os.listdir('/proc/self/fd')
+ except OSError as e:
+ if e.errno != errno.ENOENT:
+ raise
+ if fdv is not None:
+ for fd_ in fdv:
+ fd = int(fd_)
+ if (0 <= fd <= 2) or fd == but:
+ continue
+ try:
+ os.close(fd)
+ except Exception: # e.g. not KeyboardInterrupt nor SystemExit
+ pass
+
+ # fallback to closing all file descriptors in 3..MAXFD range
+ elif hasattr(os, 'closerange'):
os.closerange(3, but)
os.closerange(but + 1, MAXFD)
else:
--
2.30.2
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment