From 93b417ed1dd2b888bc9fa5d6642fe366291294fc Mon Sep 17 00:00:00 2001 From: David Lakin Date: Sun, 24 Nov 2024 20:47:30 -0500 Subject: [PATCH] [infra] Upgrade Python to 3.10.14 in base-builder & base-runner Images (#12027) > [!NOTE] > I was looking for somewhere to get feedback from maintainers about this approach to the Python 3.10 upgrade before attempting it, but the discussion surrounding a Python upgrade has been rather fragmented across many issues, PRs, and comment chains. > > For that reason, I felt it would be easier to propose with a working example and dedicated PR. #### Fixes: - #11419 - #9638 #### Supersedes: - #9532 - #11420 ## Changes The changes introduced here upgrade Python from 3.8 to 3.10.14 inside the base-builder and base-runner images. ### Base Image Changes | Image | Before Changes | After Changes | |----------------|------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **base-builder** | Compiled Python 3.8 from source using official release servers at https://www.python.org/ftp/python/. | Compiles Python 3.10.14 (the latest 3.10 release) from source using official release servers at https://www.python.org/ftp/python/. | | **base-runner** | Installed Python 3.8 from the default apt repository provided by the Ubuntu 20.04 image. | Uses a multi-stage build to copy the Python 3.10.14 interpreter compiled by the base-builder image, ensuring version sync and saving build time by re-using a pre-built version. | ## Known Impact on Projects ### 3.9 Workarounds That Can Be Removed | Project | Fix Link | |------------|----------| | dask | https://github.com/DaveLak/oss-fuzz/commit/417bbf566e3ffd962e09a50903282233b3676b69 | | docutils | https://github.com/DaveLak/oss-fuzz/commit/e4c21ffbd8baa83d97d7dd0e1f8e3e7adeae43d1 | | dovecot | https://github.com/DaveLak/oss-fuzz/commit/7ab3ab6e2604396f99398a1236ff6ddb91b04430 | | nbclassic | https://github.com/DaveLak/oss-fuzz/commit/5509b4e221ecad6136b353780daf1ceb3f457f53 | | pandas | https://github.com/DaveLak/oss-fuzz/commit/0642a7afa96d193439cc9ebd7659a0fccc7e5c1b | | pybind11 | https://github.com/DaveLak/oss-fuzz/commit/a5bbdb3dfbddd7f37dd7737cb57bb39c9ec5cc5f | | pyodbc | https://github.com/DaveLak/oss-fuzz/commit/afa2b5edc96d581329fe0405d463b026abbea3ad | | qpid-proton| https://github.com/DaveLak/oss-fuzz/commit/f5bf75628649fea5696fd8e95e1e698092ea4e74 | ### Anticipated Build Failures #### Preexisting Failures ##### Fix is Prepared | Project | Fix Link | |-----------------------|----------| | airflow | https://github.com/DaveLak/oss-fuzz/commit/60a03686280b63e88773ef9b25cd191ab8e959d0 | | ipython | https://github.com/DaveLak/oss-fuzz/commit/21ac68e9ba44e063d04b3df394330f6490081bcb | | networkx | https://github.com/DaveLak/oss-fuzz/commit/fc2f8c5481262849b586ebb8afdfdf3e6672f44f | | numpy | https://github.com/DaveLak/oss-fuzz/commit/9383c876dccad016f169d43a948c5bdc45b2894d | | tensorflow-addons | https://github.com/DaveLak/oss-fuzz/commit/eed2bea1190abd51f3c621ccc923fecb6585a67c | | django (coverage build)| https://github.com/DaveLak/oss-fuzz/commit/c724d61eae52e309726dc0d90f22bdbbecee04e7 | | proto-plus-python | https://github.com/DaveLak/oss-fuzz/commit/37d973ee8726c33678da37fb41e48d264418f097 | | dnspython | The upgraded pip version in the base-builder fixes the currently failing build. | ##### Fix Requires Upstream Changes | Project | Issue | |---------|-------| | pyvex | Currently failing on python 3.9 because `archinfo` dependency requires >=3.10. Fails after the 3.10 upgrade because [the upstream build script needs `python3.9` replaced with `python3`](https://github.com/angr/pyvex/blob/f94c95636a3800c5bbd781ecf1e3fb0c0d9feec4/fuzzing/build.sh#L19-L23). | ##### Requires More Investigation | Project | Issue | |--------------------|-------| | matplotlib | Upgrading Python & Pyinstaller does resolve the build issues, but an error in the fuzz harness is exposed and must be resolved for check_build to pass. The exception: `TypeError: Parser.non_math() takes 2 positional arguments but 4 were given" in "File "fuzz_plt.py", line 43, in TestOneInput`. | | scipy | Upgrading Python & Pyinstaller does resolve the build issues, but an error in the build step causes the build to fail. The error seems related to the linking: "/usr/bin/ld: /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x25". When `export LDFLAGS="-fuse-ld=lld"` is set, the error becomes: "`ld.lld: error: undefined symbol: __asan_report_store4`". | | pandas (Introspector only)| [This workaround in `build.sh` is the issue](https://github.com/google/oss-fuzz/blob/1515519a665756d8a50a6c46abac8b431e5462ef/projects/pandas/build.sh#L22-L32). | | pycrypto | Failing with error: "`SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats`". Seems like the issue described [here](https://stackoverflow.com/a/71019907). Pycrypto is deprecated and this is unlikely to be fixed upstream. | ## Possible Future Improvements Using the base-builder image in a multi-stage build to copy the pre- compiled Python into base-runner is effective, but feels like a workaround that may be introducing tech debt. A cleaner approach would be to extract the Python compilation into a discrete base image similar to how `base-clang` works, and use that as the multi-stage builder in images that need it. ### Fuzz Introspector Caveat Fuzz Introspector currently uses Python 3.9. While an upgrade to 3.10 is not expected to introduce any new issues, it was not tested on these changes and may require additional work. --- ## Motivation - Python [3.8 is reaching end of life in October 2024](https://devguide.python.org/versions/). - The [Scientific Python Community already encourages dropping 3.8 support](https://scientific-python.org/specs/spec-0000/). - This is evident when looking at which projects have resorted to upgrading to newer Pythons using ad-hoc workarounds (see `numpy`, `scipy`, `pandas`, etc.) - It is likely that more Python projects will begin dropping support for 3.8, further increasing the number of broken builds and ad-hoc workarounds. - Code coverage does not work on Python projects that use Python 3.10+ syntax. - Previous attempts at upgrading Python have stalled (see https://github.com/google/clusterfuzz/issues/3290#issuecomment-1699573706 & the issues linked under "Supersedes" above.) - In recognition of the fact that OSS-Fuzz maintainers are stretched thin, I thought I'd give it a shot. --------- Co-authored-by: Oliver Chang Co-authored-by: Andrew Murray --- infra/base-images/base-builder/Dockerfile | 14 ++++++----- .../base-builder/compile_python_fuzzer | 4 +-- .../base-builder/install_python.sh | 2 +- .../pysecsan/pysecsan/command_injection.py | 2 +- .../sanitizers/pysecsan/pysecsan/sanlib.py | 4 +-- infra/base-images/base-runner/Dockerfile | 22 +++++++++++++++- infra/base-images/base-runner/install_deps.sh | 4 +-- projects/configparser/Dockerfile | 2 +- projects/configparser/build.sh | 2 +- projects/django/Dockerfile | 10 +------- projects/fwupd/Dockerfile | 5 ++-- projects/nbclassic/build.sh | 2 +- projects/pffft/Dockerfile | 3 ++- .../fuzz_json_serialization.py | 25 ++++++++----------- projects/pybind11/Dockerfile | 3 +-- projects/pybind11/build.sh | 6 ++--- projects/pycrypto/Dockerfile | 4 ++- projects/pyzmq/Dockerfile | 5 ++-- projects/pyzmq/build.sh | 5 ++-- projects/six/fuzz_six.py | 3 --- 20 files changed, 68 insertions(+), 59 deletions(-) diff --git a/infra/base-images/base-builder/Dockerfile b/infra/base-images/base-builder/Dockerfile index 0c5ef2def986..64d11095b1ba 100644 --- a/infra/base-images/base-builder/Dockerfile +++ b/infra/base-images/base-builder/Dockerfile @@ -19,9 +19,9 @@ FROM gcr.io/oss-fuzz-base/base-clang COPY install_deps.sh / RUN /install_deps.sh && rm /install_deps.sh -# Build and install latest Python 3 (3.8.3). -ENV PYTHON_VERSION 3.8.3 -RUN export PYTHON_DEPS="\ +# Build and install latest Python 3.10. +ENV PYTHON_VERSION 3.10.14 +RUN PYTHON_DEPS="\ zlib1g-dev \ libncurses5-dev \ libgdbm-dev \ @@ -39,12 +39,14 @@ RUN export PYTHON_DEPS="\ tar -xvf Python-$PYTHON_VERSION.tar.xz && \ cd Python-$PYTHON_VERSION && \ ./configure --enable-optimizations --enable-shared && \ - make -j install && \ + make -j$(nproc) install && \ ldconfig && \ - ln -s /usr/bin/python3 /usr/bin/python && \ + ln -s /usr/local/bin/python3 /usr/local/bin/python && \ cd .. && \ rm -r /tmp/Python-$PYTHON_VERSION.tar.xz /tmp/Python-$PYTHON_VERSION && \ - rm -rf /usr/local/lib/python3.8/test && \ + rm -rf /usr/local/lib/python${PYTHON_VERSION%.*}/test && \ + python3 -m ensurepip && \ + python3 -m pip install --upgrade pip && \ apt-get remove -y $PYTHON_DEPS # https://github.com/google/oss-fuzz/issues/3888 diff --git a/infra/base-images/base-builder/compile_python_fuzzer b/infra/base-images/base-builder/compile_python_fuzzer index 40629a007abd..a36c05f3d1a2 100755 --- a/infra/base-images/base-builder/compile_python_fuzzer +++ b/infra/base-images/base-builder/compile_python_fuzzer @@ -32,7 +32,7 @@ if [[ $SANITIZER = *introspector* ]]; then # we enter the virtual environment in the following lines because we need # to use the same python environment that installed the fuzzer dependencies. python3 /fuzz-introspector/frontends/python/prepare_fuzz_imports.py $fuzzer_path isossfuzz - + # We must ensure python3.9, this is because we use certain # AST logic from there. # The below should probably be refined @@ -84,7 +84,7 @@ then if [[ ! -d "/pysecsan" ]]; then pushd /usr/local/lib/sanitizers/pysecsan - python3 setup.py install + python3 -m pip install . popd fi diff --git a/infra/base-images/base-builder/install_python.sh b/infra/base-images/base-builder/install_python.sh index 01403eae3c66..0d5fcb96450a 100755 --- a/infra/base-images/base-builder/install_python.sh +++ b/infra/base-images/base-builder/install_python.sh @@ -19,5 +19,5 @@ echo "ATHERIS INSTALL" unset CFLAGS CXXFLAGS # PYI_STATIC_ZLIB=1 is needed for installing pyinstaller 5.0 export PYI_STATIC_ZLIB=1 -LIBFUZZER_LIB=$( echo /usr/local/lib/clang/*/lib/x86_64-unknown-linux-gnu/libclang_rt.fuzzer_no_main.a ) pip3 install -v --no-cache-dir "atheris>=2.1.1" "pyinstaller==5.0.1" "setuptools==42.0.2" "coverage==6.3.2" +LIBFUZZER_LIB=$( echo /usr/local/lib/clang/*/lib/x86_64-unknown-linux-gnu/libclang_rt.fuzzer_no_main.a ) pip3 install -v --no-cache-dir "atheris>=2.3.0" "pyinstaller==6.10.0" "setuptools==72.1.0" "coverage==6.3.2" rm -rf /tmp/* diff --git a/infra/base-images/base-builder/sanitizers/pysecsan/pysecsan/command_injection.py b/infra/base-images/base-builder/sanitizers/pysecsan/pysecsan/command_injection.py index a6c6d09b4c2b..1d010047588f 100644 --- a/infra/base-images/base-builder/sanitizers/pysecsan/pysecsan/command_injection.py +++ b/infra/base-images/base-builder/sanitizers/pysecsan/pysecsan/command_injection.py @@ -98,7 +98,7 @@ def hook_pre_exec_os_system(cmd): 'Command injection') -def hook_pre_exec_eval(cmd): +def hook_pre_exec_eval(cmd, *args, **kwargs): """Hook for eval. Experimental atm.""" res = check_code_injection_match(cmd, check_unquoted=True) if res is not None: diff --git a/infra/base-images/base-builder/sanitizers/pysecsan/pysecsan/sanlib.py b/infra/base-images/base-builder/sanitizers/pysecsan/pysecsan/sanlib.py index 4647f47ffd5d..eead5c3979e5 100644 --- a/infra/base-images/base-builder/sanitizers/pysecsan/pysecsan/sanlib.py +++ b/infra/base-images/base-builder/sanitizers/pysecsan/pysecsan/sanlib.py @@ -22,7 +22,7 @@ import functools import subprocess import traceback -import importlib +import importlib.util from typing import Any, Callable, Optional from pysecsan import command_injection, redos, yaml_deserialization @@ -54,7 +54,7 @@ def sanitizer_log_always(msg, log_prefix=True): def is_module_present(mod_name): """Identify if module is importable.""" # pylint: disable=deprecated-method - return importlib.find_loader(mod_name) is not None + return importlib.util.find_spec(mod_name) is not None def _log_bug(bug_title): diff --git a/infra/base-images/base-runner/Dockerfile b/infra/base-images/base-runner/Dockerfile index 0a6c52a9daf7..2235281301c2 100644 --- a/infra/base-images/base-runner/Dockerfile +++ b/infra/base-images/base-runner/Dockerfile @@ -25,6 +25,11 @@ RUN cargo install rustfilt FROM gcr.io/oss-fuzz-base/base-clang AS base-clang FROM gcr.io/oss-fuzz-base/base-builder-ruby AS base-ruby +# The base builder image compiles a specific Python version. Using a multi-stage build +# to copy that same Python interpreter into the runner image saves build time and keeps +# the Python versions in sync. +FROM gcr.io/oss-fuzz-base/base-builder AS base-builder + # Real image that will be used later. FROM gcr.io/oss-fuzz-base/base-image @@ -36,6 +41,18 @@ COPY --from=base-clang /usr/local/bin/llvm-cov \ /usr/local/bin/llvm-symbolizer \ /usr/local/bin/ +# Copy the pre-compiled Python binaries and libraries +COPY --from=base-builder /usr/local/bin/python3.10 /usr/local/bin/python3.10 +COPY --from=base-builder /usr/local/lib/libpython3.10.so.1.0 /usr/local/lib/libpython3.10.so.1.0 +COPY --from=base-builder /usr/local/include/python3.10 /usr/local/include/python3.10 +COPY --from=base-builder /usr/local/lib/python3.10 /usr/local/lib/python3.10 +COPY --from=base-builder /usr/local/bin/pip3 /usr/local/bin/pip3 + +# Create symbolic links to ensure compatibility +RUN ldconfig && \ + ln -s /usr/local/bin/python3.10 /usr/local/bin/python3 && \ + ln -s /usr/local/bin/python3.10 /usr/local/bin/python + COPY install_deps.sh / RUN /install_deps.sh && rm /install_deps.sh @@ -46,8 +63,11 @@ RUN git clone https://chromium.googlesource.com/chromium/src/tools/code_coverage cd /opt/code_coverage && \ git checkout edba4873b5e8a390e977a64c522db2df18a8b27d && \ pip3 install wheel && \ + # If version "Jinja2==2.10" is in requirements.txt, bump it to a patch version that + # supports upgrading its MarkupSafe dependency to a Python 3.10 compatible release: + sed -i 's/Jinja2==2.10/Jinja2==2.10.3/' requirements.txt && \ pip3 install -r requirements.txt && \ - pip3 install MarkupSafe==0.23 && \ + pip3 install MarkupSafe==2.0.1 && \ pip3 install coverage==6.3.2 # Default environment options for various sanitizers. diff --git a/infra/base-images/base-runner/install_deps.sh b/infra/base-images/base-runner/install_deps.sh index ef12cde0061a..fc0569b339ad 100755 --- a/infra/base-images/base-runner/install_deps.sh +++ b/infra/base-images/base-runner/install_deps.sh @@ -20,12 +20,10 @@ apt-get update && apt-get install -y \ binutils \ file \ + ca-certificates \ fonts-dejavu \ git \ libcap2 \ - python3 \ - python3-pip \ - python3-setuptools \ rsync \ unzip \ wget \ diff --git a/projects/configparser/Dockerfile b/projects/configparser/Dockerfile index 3928e4b6a7de..b328b69d6370 100644 --- a/projects/configparser/Dockerfile +++ b/projects/configparser/Dockerfile @@ -13,7 +13,7 @@ # See the License for the specific language governing permissions and # limitations under the License. FROM gcr.io/oss-fuzz-base/base-builder-python -RUN pip3 install --upgrade pip +RUN python -m pip install 'setuptools~=69.0' RUN git clone https://github.com/jaraco/configparser/ configparser COPY *.sh *py $SRC/ WORKDIR $SRC/configparser diff --git a/projects/configparser/build.sh b/projects/configparser/build.sh index dd11fafc62bb..50be783e9df1 100644 --- a/projects/configparser/build.sh +++ b/projects/configparser/build.sh @@ -14,7 +14,7 @@ # limitations under the License. # ################################################################################ -pip3 install . +python -m pip install . # Build fuzzers in $OUT. for fuzzer in $(find $SRC -name 'fuzz_*.py'); do compile_python_fuzzer $fuzzer diff --git a/projects/django/Dockerfile b/projects/django/Dockerfile index 9f220da0fdc6..35593e91f505 100644 --- a/projects/django/Dockerfile +++ b/projects/django/Dockerfile @@ -16,15 +16,7 @@ FROM gcr.io/oss-fuzz-base/base-builder-python -RUN apt-get update -y \ - && apt-get install -y libgdal26 software-properties-common \ - && add-apt-repository -y ppa:deadsnakes/ppa \ - && apt-get install -y python3.10 python3.10-dev \ - && ln --force -s /usr/bin/python3.10 /usr/local/bin/python3 \ - && curl -sS https://bootstrap.pypa.io/get-pip.py | python3 \ - && python3 -m pip install -v --no-cache-dir "atheris>=2.1.1" "pyinstaller==5.0.1" "coverage==6.3.2" \ - && rm -rf /var/lib/apt/lists/* - +RUN python3 -m pip install cython RUN git clone --depth 1 https://github.com/django/django.git RUN git clone --depth 1 https://github.com/django/django-fuzzers.git diff --git a/projects/fwupd/Dockerfile b/projects/fwupd/Dockerfile index fd0d9011f8ee..45daa800fed5 100644 --- a/projects/fwupd/Dockerfile +++ b/projects/fwupd/Dockerfile @@ -15,9 +15,8 @@ ################################################################################ FROM gcr.io/oss-fuzz-base/base-builder -RUN apt-get update -RUN apt-get install -y pkg-config zlib1g-dev libffi-dev liblzma-dev libcbor-dev -RUN pip3 install -U meson ninja +RUN apt-get update && apt-get install -y pkg-config zlib1g-dev libffi-dev liblzma-dev libcbor-dev +RUN python3 -m pip install -U jinja2 packaging meson ninja RUN git clone --depth 1 https://github.com/fwupd/fwupd.git fwupd WORKDIR . COPY build.sh $SRC/ diff --git a/projects/nbclassic/build.sh b/projects/nbclassic/build.sh index 393171058aad..6528079d0e8f 100644 --- a/projects/nbclassic/build.sh +++ b/projects/nbclassic/build.sh @@ -16,5 +16,5 @@ ################################################################################ pip3 install . for fuzzer in $(find $SRC -name 'fuzz_*.py'); do - compile_python_fuzzer $fuzzer --add-data $SRC/jsonschema_specifications/jsonschema_specifications/schemas:jsonschema_specifications/schemas --add-data /usr/local/lib/python3.8/site-packages/jupyter_events/schemas:jupyter_events/schemas + compile_python_fuzzer $fuzzer --add-data $SRC/jsonschema_specifications/jsonschema_specifications/schemas:jsonschema_specifications/schemas --add-data /usr/local/lib/python3.10/site-packages/jupyter_events/schemas:jupyter_events/schemas done diff --git a/projects/pffft/Dockerfile b/projects/pffft/Dockerfile index f0faf12f4bcd..221d63b103d7 100644 --- a/projects/pffft/Dockerfile +++ b/projects/pffft/Dockerfile @@ -15,7 +15,8 @@ ################################################################################ FROM gcr.io/oss-fuzz-base/base-builder -RUN apt-get update && apt-get install -y mercurial python-numpy python +RUN apt-get update && apt-get install -y mercurial +RUN python -m pip install numpy RUN git clone https://bitbucket.org/jpommier/pffft $SRC/pffft WORKDIR pffft COPY build.sh $SRC diff --git a/projects/proto-plus-python/fuzz_json_serialization.py b/projects/proto-plus-python/fuzz_json_serialization.py index 1237046915cd..cf8d80ea2caf 100644 --- a/projects/proto-plus-python/fuzz_json_serialization.py +++ b/projects/proto-plus-python/fuzz_json_serialization.py @@ -18,32 +18,29 @@ import proto from google.protobuf.json_format import ParseError + +class FuzzMsg(proto.Message): + val1 = proto.Field(proto.FLOAT, number=1) + val2 = proto.Field(proto.INT32, number=2) + val3 = proto.Field(proto.BOOL, number=3) + val4 = proto.Field(proto.STRING, number=4) + + def TestOneInput(data): fdp = atheris.FuzzedDataProvider(data) - class FuzzMsg(proto.Message): - val1 = proto.Field(proto.FLOAT, number=1) - val2 = proto.Field(proto.INT32, number=2) - val3 = proto.Field(proto.BOOL, number=3) - val4 = proto.Field(proto.STRING, number=4) - try: s = FuzzMsg.from_json(fdp.ConsumeUnicodeNoSurrogates(sys.maxsize)) FuzzMsg.to_json(s) - except ParseError: - pass - except TypeError: - pass - except RecursionError: - pass + except (ParseError, TypeError, RecursionError): + return def main(): atheris.instrument_all() - atheris.Setup(sys.argv, TestOneInput, enable_python_coverage=True) + atheris.Setup(sys.argv, TestOneInput) atheris.Fuzz() if __name__ == "__main__": main() - diff --git a/projects/pybind11/Dockerfile b/projects/pybind11/Dockerfile index f2fa2091d1ad..9011e8bf2c81 100644 --- a/projects/pybind11/Dockerfile +++ b/projects/pybind11/Dockerfile @@ -16,8 +16,7 @@ FROM gcr.io/oss-fuzz-base/base-builder RUN apt-get update && \ - apt-get install -y python-is-python3 build-essential pip python3-dev -RUN python3 -m pip install --upgrade pip + apt-get install -y build-essential RUN git clone https://github.com/pybind/pybind11 COPY build.sh *_fuzzer.cc $SRC/ diff --git a/projects/pybind11/build.sh b/projects/pybind11/build.sh index 75fef7b06ede..b2eaffe6fed9 100755 --- a/projects/pybind11/build.sh +++ b/projects/pybind11/build.sh @@ -20,13 +20,13 @@ cmake -S . -B build -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON cmake --build build -j4 python3 -m pip install . -cp /usr/local/lib/libpython3.8.so.1.0 $OUT/ +cp /usr/local/lib/libpython3.10.so.1.0 $OUT/ for f in $SRC/*_fuzzer.cc; do fuzzer=$(basename "$f" _fuzzer.cc) $CXX $CXXFLAGS \ - -I$SRC/pybind11/include -isystem /usr/local/include/python3.8 \ + -I$SRC/pybind11/include -isystem /usr/local/include/python3.10 \ $SRC/${fuzzer}_fuzzer.cc -o $OUT/${fuzzer}_fuzzer \ - /usr/local/lib/libpython3.8.so.1.0 \ + /usr/local/lib/libpython3.10.so.1.0 \ $LIB_FUZZING_ENGINE -lpthread patchelf --set-rpath '$ORIGIN/' $OUT/${fuzzer}_fuzzer done diff --git a/projects/pycrypto/Dockerfile b/projects/pycrypto/Dockerfile index 3ea6ad6f716d..52946c2ccd00 100644 --- a/projects/pycrypto/Dockerfile +++ b/projects/pycrypto/Dockerfile @@ -14,7 +14,9 @@ # ################################################################################ -FROM gcr.io/oss-fuzz-base/base-builder-python +# Held back because of github.com/google/oss-fuzz/pull/12027 +# Please fix failure and upgrade if possible. +FROM gcr.io/oss-fuzz-base/base-builder-python@sha256:d8fe5e2a6a96723f393de413c48d9455a5124995b2349a2e4d6b9abecf99d6d5 RUN git clone https://github.com/pycrypto/pycrypto COPY build.sh *.py $SRC/ WORKDIR pycrypto diff --git a/projects/pyzmq/Dockerfile b/projects/pyzmq/Dockerfile index 710555f0a40a..bb16d8bf1cbd 100644 --- a/projects/pyzmq/Dockerfile +++ b/projects/pyzmq/Dockerfile @@ -13,7 +13,8 @@ # See the License for the specific language governing permissions and # limitations under the License. FROM gcr.io/oss-fuzz-base/base-builder-python -RUN pip3 install --upgrade pip cython -RUN git clone https://github.com/zeromq/pyzmq pyzmq +RUN apt-get update && apt-get install -y libzmq3-dev +RUN python -m pip install cython +RUN git clone --depth 1 --branch main https://github.com/zeromq/pyzmq pyzmq COPY *.sh *py $SRC/ WORKDIR $SRC/pyzmq diff --git a/projects/pyzmq/build.sh b/projects/pyzmq/build.sh index 83cdadf8cadb..ac81d920ad6c 100644 --- a/projects/pyzmq/build.sh +++ b/projects/pyzmq/build.sh @@ -14,7 +14,8 @@ # limitations under the License. # ################################################################################ -pip3 install . +python -m pip install . + for fuzzer in $(find $SRC -name 'fuzz_*.py'); do - compile_python_fuzzer $fuzzer + compile_python_fuzzer $fuzzer --collect-all="pyzmq" done diff --git a/projects/six/fuzz_six.py b/projects/six/fuzz_six.py index cd3746ed3bd9..8e237283e93b 100644 --- a/projects/six/fuzz_six.py +++ b/projects/six/fuzz_six.py @@ -55,9 +55,6 @@ def TestOneInput(data): except (TypeError, UnicodeDecodeError): pass - six.moves.html_parser.HTMLParser().unescape( - fdp.ConsumeUnicodeNoSurrogates(fdp.ConsumeIntInRange(1, 1024))) - def main(): atheris.instrument_all()