OpenSSL with FIPS enabled causes test_hashlib failures

[ghmj2417]

Bug report

Bug description:

We are using OpenSSL 3.3.1 with FIPS enabled. When compiling Python version 3.13.1 with --enable-optimizations, test_hashlib will fail with the following.

...     fetching http://www.pythontest.net/hashlib/blake2b.txt ...
..      fetching http://www.pythontest.net/hashlib/blake2s.txt ...
.............ss.................        fetching http://www.pythontest.net/hashlib/sha3_224.txt ...
..      fetching http://www.pythontest.net/hashlib/sha3_256.txt ...
..      fetching http://www.pythontest.net/hashlib/sha3_384.txt ...
..      fetching http://www.pythontest.net/hashlib/sha3_512.txt ...
.....   fetching http://www.pythontest.net/hashlib/shake_128.txt ...
.       fetching http://www.pythontest.net/hashlib/shake_256.txt ...
.......................EE...E
======================================================================
ERROR: test_usedforsecurity_false (__main__.HashLibTestCase.test_usedforsecurity_false)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/Python-3.13.1/Lib/test/test_hashlib.py", line 246, in test_usedforsecurity_false
    self._hashlib.new("md5", usedforsecurity=False)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_hashlib.UnsupportedDigestmodError: [digital envelope routines] unsupported

======================================================================
ERROR: test_usedforsecurity_true (__main__.HashLibTestCase.test_usedforsecurity_true)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/Python-3.13.1/Lib/test/test_hashlib.py", line 235, in test_usedforsecurity_true
    self._hashlib.new("md5", usedforsecurity=True)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_hashlib.UnsupportedDigestmodError: [digital envelope routines] unsupported

======================================================================
ERROR: test_scrypt (__main__.KDFTests.test_scrypt)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/Python-3.13.1/Lib/test/test_hashlib.py", line 1122, in test_scrypt
    result = hashlib.scrypt(password, salt=salt, n=n, r=r, p=p)
ValueError: [digital envelope routines] unsupported

----------------------------------------------------------------------
Ran 78 tests in 24.957s

FAILED (errors=3, skipped=2)

I believe there was a change with 3.12 where test failures no longer fail silently so it would make sense why our Python 3.11 build does not fail.

I did some digging and I found the following PRs and issue which I believe pertain to what I am seeing:

• gh-127298: When in FIPS mode ensure builtin hashes check for usedforsecurity=False #127301
• gh-127298: Refactor test_hashlib for better usedforsecurity & openssl fips mode env support. #127492
• Ensure builtin hashlib implementations honor usedforsecurity=True when _hashlib is in FIPS mode #127298

CPython versions tested on:

3.13

Operating systems tested on:

Linux