Software Heritage fallback fixes #6036

rjbou · 2024-06-19T17:47:18Z

Fix archive cooking request for wget, the wrong option was given
Fix SWH liveness check, no POST request. It is a hidden bug, it was failing with no error code for curl and as wget had the wrong POST request option it wasn't showing.
Update SWH API request: vault/directory is deprecated in favor to vault/flat
Some doc
fixes opam does not wait for the Software Heritage fallback archives to be "cooked" #5721

dra27

A couple of questions, and I wonder if we may as well strictly follow the protocol for vault/flat, given that the code makes it relatively easy for us to do so.

src/repository/opamDownload.ml

dra27

LGTM - I'm puzzled by the slightly arcane way swh seems to specify vault/flat, but that doesn't need to affect the code here!

rjbou · 2024-11-15T17:00:43Z

Updated!
Some outputs:

wget & not cooked

> ./opam install  -vvv --debug -y swh-wget-not-cooked
[...]

[...]
The following actions will be performed:
=== install 1 package
  ∗ swh-wget-not-cooked 1

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
Processing  1/3: [swh-wget-not-cooked.1: http]
+ /usr/bin/wget "-t" "3" "-O" "/tmp/opam-1102792-e86235/fkdjhfskjfhsjkdhfake.o.r.g.part" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://fkdjhfskjfhsjkdhfake.o.r.g/"
- --2024-11-15 16:08:51--  https://fkdjhfskjfhsjkdhfake.o.r.g/
- Resolving fkdjhfskjfhsjkdhfake.o.r.g (fkdjhfskjfhsjkdhfake.o.r.g)... failed: Name or service not known.
- wget: unable to resolve host address ‘fkdjhfskjfhsjkdhfake.o.r.g’
+ /usr/bin/wget "-t" "3" "-O" "-" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/ping/"
- --2024-11-15 16:08:51--  https://archive.softwareheritage.org/api/1/ping/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Date: Fri, 15 Nov 2024 15:08:51 GMT
-   Content-Type: application/json
-   Content-Length: 6
-   Vary: Accept, origin, Cookie
-   Allow: HEAD, GET, OPTIONS, OPTIONS
-   X-RateLimit-Limit: 120
-   X-RateLimit-Remaining: 113
-   X-RateLimit-Reset: 1731685708
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 47910447
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Accept-Ranges: bytes
-   Connection: keep-alive
- Length: 6 [application/json]
- Saving to: ‘STDOUT’
- 
-      0K                                                       100% 8.32M=0s
- 
- 2024-11-15 16:08:51 (8.32 MB/s) - written to stdout [6/6]
- 
- "pong"
00:14.316  CURL(SWHID)            API is working
Source https://fkdjhfskjfhsjkdhfake.o.r.g/ is not available. Do you want to try to retrieve it from Software Heritage cache
(https://www.softwareheritage.org)? It may take few minutes. [y/n] y
00:14.317  CURL(SWHID)            SWH fallback for swh-wget-not-cooked.1 with swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331
Processing  1/3:
+ /usr/bin/wget "--content-on-error" "-t" "3" "-O" "/opamroot/log/swh-out-1102792-c8bbc0" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/"
- --2024-11-15 16:08:51--  https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 404 Not Found
-   Date: Fri, 15 Nov 2024 15:08:51 GMT
-   Content-Type: application/json
-   Content-Length: 121
-   Expires: Fri, 15 Nov 2024 15:08:51 GMT
-   Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
-   Vary: Accept, origin, Cookie
-   Allow: POST, OPTIONS, GET
-   X-RateLimit-Limit: 60
-   X-RateLimit-Remaining: 59
-   X-RateLimit-Reset: 1731683391
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 50336527
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Connection: keep-alive
- Saving to: ‘/opamroot/log/swh-out-1102792-c8bbc0’
- 
-      0K                                                       100%  224M=0s
- 
- 2024-11-15 16:08:51 ERROR 404: Not Found.
- 
00:14.528  CURL(SWHID)            swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331 is uncooked, request cooking
+ /usr/bin/wget "--method" "POST" "-t" "3" "-O" "-" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/"
- --2024-11-15 16:08:51--  https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Date: Fri, 15 Nov 2024 15:08:52 GMT
-   Content-Type: application/json
-   Content-Length: 240
-   Expires: Fri, 15 Nov 2024 15:08:52 GMT
-   Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
-   Vary: Accept, origin, Cookie
-   Allow: POST, OPTIONS, GET
-   X-RateLimit-Limit: 120
-   X-RateLimit-Remaining: 119
-   X-RateLimit-Reset: 1731686931
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 53773176
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Accept-Ranges: bytes
-   Connection: keep-alive
- Length: 240 [application/json]
- Saving to: ‘STDOUT’
- 
-      0K                                                       100%  938M=0s
- 
- 2024-11-15 16:08:52 (938 MB/s) - written to stdout [240/240]
- 
- {"fetch_url":"https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/raw/","progress_message":null,"id":416944911,"status":"new","swhid":"swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331"}
00:14.867  CURL(SWHID)            swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331 is cooking...
+ /usr/bin/wget "-t" "3" "-O" "-" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/"
- --2024-11-15 16:09:02--  https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Date: Fri, 15 Nov 2024 15:09:02 GMT
-   Content-Type: application/json
-   Content-Length: 240
-   Expires: Fri, 15 Nov 2024 15:09:02 GMT
-   Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
-   Vary: Accept, origin, Cookie
-   Allow: POST, OPTIONS, GET
-   X-RateLimit-Limit: 60
-   X-RateLimit-Remaining: 58
-   X-RateLimit-Reset: 1731683391
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 45750401
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Accept-Ranges: bytes
-   Connection: keep-alive
- Length: 240 [application/json]
- Saving to: ‘STDOUT’
- 
-      0K                                                       100%  288M=0s
- 
- 2024-11-15 16:09:02 (288 MB/s) - written to stdout [240/240]
- 
- {"fetch_url":"https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/raw/","progress_message":null,"id":416944911,"status":"new","swhid":"swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331"}
00:25.063  CURL(SWHID)            swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331 is cooking...
+ /usr/bin/wget "-t" "3" "-O" "-" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/"
- --2024-11-15 16:09:12--  https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Date: Fri, 15 Nov 2024 15:09:12 GMT
-   Content-Type: application/json
-   Content-Length: 240
-   Expires: Fri, 15 Nov 2024 15:09:12 GMT
-   Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
-   Vary: Accept, origin, Cookie
-   Allow: GET, POST, OPTIONS
-   X-RateLimit-Limit: 60
-   X-RateLimit-Remaining: 57
-   X-RateLimit-Reset: 1731683391
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 53283606
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Accept-Ranges: bytes
-   Connection: keep-alive
- Length: 240 [application/json]
- Saving to: ‘STDOUT’
- 
-      0K                                                       100%  300M=0s
- 
- 2024-11-15 16:09:12 (300 MB/s) - written to stdout [240/240]
- 
- {"fetch_url":"https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/raw/","progress_message":null,"id":416944911,"status":"new","swhid":"swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331"}
00:35.263  CURL(SWHID)            swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331 is cooking...
+ /usr/bin/wget "-t" "3" "-O" "-" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/"
- --2024-11-15 16:09:22--  https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Date: Fri, 15 Nov 2024 15:09:22 GMT
-   Content-Type: application/json
-   Content-Length: 240
-   Expires: Fri, 15 Nov 2024 15:09:22 GMT
-   Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
-   Vary: Accept, origin, Cookie
-   Allow: GET, POST, OPTIONS
-   X-RateLimit-Limit: 60
-   X-RateLimit-Remaining: 56
-   X-RateLimit-Reset: 1731683391
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 55281750
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Accept-Ranges: bytes
-   Connection: keep-alive
- Length: 240 [application/json]
- Saving to: ‘STDOUT’
- 
-      0K                                                       100%  245M=0s
- 
- 2024-11-15 16:09:22 (245 MB/s) - written to stdout [240/240]
- 
- {"fetch_url":"https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/raw/","progress_message":null,"id":416944911,"status":"new","swhid":"swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331"}
00:45.481  CURL(SWHID)            swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331 is cooking...
+ /usr/bin/wget "-t" "3" "-O" "-" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/"
- --2024-11-15 16:09:32--  https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Date: Fri, 15 Nov 2024 15:09:32 GMT
-   Content-Type: application/json
-   Content-Length: 240
-   Expires: Fri, 15 Nov 2024 15:09:32 GMT
-   Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
-   Vary: Accept, origin, Cookie
-   Allow: OPTIONS, POST, GET
-   X-RateLimit-Limit: 60
-   X-RateLimit-Remaining: 55
-   X-RateLimit-Reset: 1731683391
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 52529512
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Accept-Ranges: bytes
-   Connection: keep-alive
- Length: 240 [application/json]
- Saving to: ‘STDOUT’
- 
-      0K                                                       100%  272M=0s
- 
- 2024-11-15 16:09:33 (272 MB/s) - written to stdout [240/240]
- 
- {"fetch_url":"https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/raw/","progress_message":null,"id":416944911,"status":"new","swhid":"swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331"}
00:55.864  CURL(SWHID)            swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331 is cooking...
+ /usr/bin/wget "-t" "3" "-O" "-" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/"
- --2024-11-15 16:09:43--  https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Date: Fri, 15 Nov 2024 15:09:43 GMT
-   Content-Type: application/json
-   Content-Length: 241
-   Expires: Fri, 15 Nov 2024 15:09:43 GMT
-   Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
-   Vary: Accept, origin, Cookie
-   Allow: GET, OPTIONS, POST
-   X-RateLimit-Limit: 60
-   X-RateLimit-Remaining: 54
-   X-RateLimit-Reset: 1731683391
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 39989269
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Accept-Ranges: bytes
-   Connection: keep-alive
- Length: 241 [application/json]
- Saving to: ‘STDOUT’
- 
-      0K                                                       100% 1.18G=0s
- 
- 2024-11-15 16:09:43 (1.18 GB/s) - written to stdout [241/241]
- 
- {"fetch_url":"https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/raw/","progress_message":null,"id":416944911,"status":"done","swhid":"swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331"}
01:06.211  CURL(SWHID)            Downloading swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331 for swh-wget-not-cooked.1
+ /usr/bin/wget "-t" "3" "-O" "/tmp/opam-1102792-70a205/6394e653c9dfa1be07250529c461dcd7983cd331.part" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/raw/"
- --2024-11-15 16:09:43--  https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6394e653c9dfa1be07250529c461dcd7983cd331/raw/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 302 Found
-   Date: Fri, 15 Nov 2024 15:09:43 GMT
-   Content-Type: text/html; charset=utf-8
-   Content-Length: 0
-   Location: https://swhvaultstorage.blob.core.windows.net/contents-uncompressed/f2391ef00cb8cab822ba66e4f9bdb0f782b84bfd?se=2024-11-16T15%3A09%3A43Z&sp=r&sv=2024-11-04&sr=b&rscd=attachment%3B%20filename%3D%22swh_1_dir_6394e653c9dfa1be07250529c461dcd7983cd331.tar.gz%22&sig=3N2KiPjDIRFVfpGo6PLwr8syJab5Ye7Q8jKJW66G0oM%3D
-   Vary: Accept, origin, Cookie
-   Allow: OPTIONS, GET, OPTIONS, HEAD
-   X-RateLimit-Limit: 120
-   X-RateLimit-Remaining: 112
-   X-RateLimit-Reset: 1731685708
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 52431141
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Connection: keep-alive
- Location: https://swhvaultstorage.blob.core.windows.net/contents-uncompressed/f2391ef00cb8cab822ba66e4f9bdb0f782b84bfd?se=2024-11-16T15%3A09%3A43Z&sp=r&sv=2024-11-04&sr=b&rscd=attachment%3B%20filename%3D%22swh_1_dir_6394e653c9dfa1be07250529c461dcd7983cd331.tar.gz%22&sig=3N2KiPjDIRFVfpGo6PLwr8syJab5Ye7Q8jKJW66G0oM%3D [following]
- --2024-11-15 16:09:43--  https://swhvaultstorage.blob.core.windows.net/contents-uncompressed/f2391ef00cb8cab822ba66e4f9bdb0f782b84bfd?se=2024-11-16T15%3A09%3A43Z&sp=r&sv=2024-11-04&sr=b&rscd=attachment%3B%20filename%3D%22swh_1_dir_6394e653c9dfa1be07250529c461dcd7983cd331.tar.gz%22&sig=3N2KiPjDIRFVfpGo6PLwr8syJab5Ye7Q8jKJW66G0oM%3D
- Resolving swhvaultstorage.blob.core.windows.net (swhvaultstorage.blob.core.windows.net)... 20.209.11.33
- Connecting to swhvaultstorage.blob.core.windows.net (swhvaultstorage.blob.core.windows.net)|20.209.11.33|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Content-Length: 15559
-   Content-Type: application/octet-stream
-   Content-MD5: 7HYlM4kpCKEi2SPr0xMKyA==
-   Last-Modified: Fri, 15 Nov 2024 15:09:38 GMT
-   Accept-Ranges: bytes
-   ETag: "0x8DD058785ED331E"
-   Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
-   x-ms-request-id: d6c83cff-a01e-0017-3b70-3757dc000000
-   x-ms-version: 2024-11-04
-   x-ms-creation-time: Fri, 15 Nov 2024 15:09:38 GMT
-   x-ms-lease-status: unlocked
-   x-ms-lease-state: available
-   x-ms-blob-type: BlockBlob
-   Content-Disposition: attachment; filename="swh_1_dir_6394e653c9dfa1be07250529c461dcd7983cd331.tar.gz"
-   x-ms-server-encrypted: true
-   Access-Control-Allow-Origin: *
-   Date: Fri, 15 Nov 2024 15:09:43 GMT
- Length: 15559 (15K) [application/octet-stream]
- Saving to: ‘/tmp/opam-1102792-70a205/6394e653c9dfa1be07250529c461dcd7983cd331.part’
- 
-      0K .......... .....                                      100% 1.31G=0s
- 
- 2024-11-15 16:09:44 (1.31 GB/s) - ‘/tmp/opam-1102792-70a205/6394e653c9dfa1be07250529c461dcd7983cd331.part’ saved [15559/15559]
- 
+ /usr/bin/tar "xfz" "/tmp/opam-1102792-70a205/6394e653c9dfa1be07250529c461dcd7983cd331" "-C" "/tmp/opam-1102792-b35983"
⬇ retrieved swh-wget-not-cooked.1  (SWH fallback)
[...]

cURL & not cooked

> ./opam install  -vvv --debug -y swh-curl-not-cooked
[...]
The following actions will be performed:
=== install 1 package
  ∗ swh-curl-not-cooked 1

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
Processing  1/3: [swh-curl-not-cooked.1: http]
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "/tmp/opam-360556-43cf30/i-am-not-available-fake.o.r.g.part" "--" "https://i-am-not-available-fake.o.r.g/"
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: i-am-not-available-fake.o.r.g
- Warning: Problem : timeout. Will retry in 2 seconds. 3 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: i-am-not-available-fake.o.r.g
- Warning: Problem : timeout. Will retry in 2 seconds. 2 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: i-am-not-available-fake.o.r.g
- Warning: Problem : timeout. Will retry in 2 seconds. 1 retries left.
- 000
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: i-am-not-available-fake.o.r.g
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "-" "--" "https://archive.softwareheritage.org/api/1/ping/"
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
100     6  100     6    0     0     49      0 --:--:-- --:--:-- --:--:--    50
- "pong"200
00:17.873  CURL(SWHID)            API is working
Source https://i-am-not-available-fake.o.r.g/ is not available. Do you want to try to retrieve it from
Software Heritage cache (https://www.softwareheritage.org)? It may take few minutes. [y/n] y
00:17.873  CURL(SWHID)            SWH fallback for swh-curl-not-cooked.1 with swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b
Processing  1/3:
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "/opamroot/log/swh-out-360556-b9353e" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b/"
- 404
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
100   121  100   121    0     0    741      0 --:--:-- --:--:-- --:--:--   746
00:18.065  CURL(SWHID)            swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b is uncooked, request cooking
+ /usr/bin/curl "--request" "POST" "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "-" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b/"
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
100   240  100   240    0     0    863      0 --:--:-- --:--:-- --:--:--   866
- {"fetch_url":"https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b/raw/","progress_message":null,"id":416941408,"status":"new","swhid":"swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b"}200
00:18.371  CURL(SWHID)            swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b is cooking...
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "-" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b/"
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
100   255  100   255    0     0   1735      0 --:--:-- --:--:-- --:--:--  1734
- {"fetch_url":"https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b/raw/","progress_message":"Processing...","id":416941408,"status":"pending","swhid":"swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b"}200
00:28.550  CURL(SWHID)            swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b is cooking...
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "-" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b/"
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
100   255  100   255    0     0   1850      0 --:--:-- --:--:-- --:--:--  1861
- {"fetch_url":"https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b/raw/","progress_message":"Processing...","id":416941408,"status":"pending","swhid":"swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b"}200
00:38.718  CURL(SWHID)            swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b is cooking...
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "-" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b/"
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
100   241  100   241    0     0   1726      0 --:--:-- --:--:-- --:--:--  1733
- {"fetch_url":"https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b/raw/","progress_message":null,"id":416941408,"status":"done","swhid":"swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b"}200
00:48.886  CURL(SWHID)            Downloading swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b for swh-curl-not-cooked.1
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "/tmp/opam-360556-ffbc05/7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b.part" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b/raw/"
- 200
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 12495  100 12495    0     0  29676      0 --:--:-- --:--:-- --:--:-- 29676
+ /usr/bin/tar "xfz" "/tmp/opam-360556-ffbc05/7a79aa8ded5a648e6a3ed19ee3a64e1a32c8672b" "-C" "/tmp/opam-360556-0a2c9e"
⬇ retrieved swh-curl-not-cooked.1  (SWH fallback)
[...]

wget & cooked

> ./opam install  -vvv --debug -y swh-wget-cooked
[...]
The following actions will be performed:
=== install 1 package
  ∗ swh-wget-cooked 1

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
Processing  1/3: [swh-wget-cooked.1: http]
+ /usr/bin/wget "-t" "3" "-O" "/tmp/opam-1100962-6df63b/fkdjhfskjfhsjkdhfake.o.r.g.part" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://fkdjhfskjfhsjkdhfake.o.r.g/"
- --2024-11-15 16:05:51--  https://fkdjhfskjfhsjkdhfake.o.r.g/
- Resolving fkdjhfskjfhsjkdhfake.o.r.g (fkdjhfskjfhsjkdhfake.o.r.g)... failed: Name or service not known.
- wget: unable to resolve host address ‘fkdjhfskjfhsjkdhfake.o.r.g’
+ /usr/bin/wget "-t" "3" "-O" "-" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/ping/"
- --2024-11-15 16:05:51--  https://archive.softwareheritage.org/api/1/ping/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Date: Fri, 15 Nov 2024 15:05:51 GMT
-   Content-Type: application/json
-   Content-Length: 6
-   Vary: Accept, origin, Cookie
-   Allow: OPTIONS, GET, OPTIONS, HEAD
-   X-RateLimit-Limit: 120
-   X-RateLimit-Remaining: 115
-   X-RateLimit-Reset: 1731685708
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 54232414
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Accept-Ranges: bytes
-   Connection: keep-alive
- Length: 6 [application/json]
- Saving to: ‘STDOUT’
- 
-      0K                                                       100% 22.9M=0s
- 
- 2024-11-15 16:05:51 (22.9 MB/s) - written to stdout [6/6]
- 
- "pong"
00:13.858  CURL(SWHID)            API is working
Source https://fkdjhfskjfhsjkdhfake.o.r.g/ is not available. Do you want to try to retrieve it from
Software Heritage cache (https://www.softwareheritage.org)? It may take few minutes. [y/n] y
00:13.858  CURL(SWHID)            SWH fallback for swh-wget-cooked.1 with swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4
Processing  1/3:
+ /usr/bin/wget "--content-on-error" "-t" "3" "-O" "/opamroot/log/swh-out-1100962-7f8485" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4/"
- --2024-11-15 16:05:51--  https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Date: Fri, 15 Nov 2024 15:05:51 GMT
-   Content-Type: application/json
-   Content-Length: 241
-   Expires: Fri, 15 Nov 2024 15:05:51 GMT
-   Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
-   Vary: Accept, origin, Cookie
-   Allow: POST, OPTIONS, GET
-   X-RateLimit-Limit: 60
-   X-RateLimit-Remaining: 59
-   X-RateLimit-Reset: 1731683211
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 47910295
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Accept-Ranges: bytes
-   Connection: keep-alive
- Length: 241 [application/json]
- Saving to: ‘/opamroot/log/swh-out-1100962-7f8485’
- 
-      0K                                                       100%  143M=0s
- 
- 2024-11-15 16:05:51 (143 MB/s) - ‘/opamroot/log/swh-out-1100962-7f8485’ saved [241/241]
- 
00:14.101  CURL(SWHID)            swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4 is cooked, requesting url
00:14.101  CURL(SWHID)            Downloading swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4 for swh-wget-cooked.1
+ /usr/bin/wget "-t" "3" "-O" "/tmp/opam-1100962-3d90aa/6b700f4b287aee509adbc723d0303098188604f4.part" "--server-response" "-U" "opam/2.4.0~alpha1~dev" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4/raw/"
- --2024-11-15 16:05:51--  https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4/raw/
- Resolving archive.softwareheritage.org (archive.softwareheritage.org)... 128.93.166.15
- Connecting to archive.softwareheritage.org (archive.softwareheritage.org)|128.93.166.15|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 302 Found
-   Date: Fri, 15 Nov 2024 15:05:52 GMT
-   Content-Type: text/html; charset=utf-8
-   Content-Length: 0
-   Location: https://swhvaultstorage.blob.core.windows.net/contents-uncompressed/94f715535502200f7a4ede59a85d22dda9fc3510?se=2024-11-16T15%3A05%3A52Z&sp=r&sv=2024-11-04&sr=b&rscd=attachment%3B%20filename%3D%22swh_1_dir_6b700f4b287aee509adbc723d0303098188604f4.tar.gz%22&sig=9DEL7O0yEMhxz3Wgvo3QgJ6180mkDIjuV4rKoBCTRTc%3D
-   Vary: Accept, origin, Cookie
-   Allow: OPTIONS, OPTIONS, GET, HEAD
-   X-RateLimit-Limit: 120
-   X-RateLimit-Remaining: 114
-   X-RateLimit-Reset: 1731685708
-   X-Frame-Options: DENY
-   X-Content-Type-Options: nosniff
-   Referrer-Policy: same-origin
-   Cross-Origin-Opener-Policy: same-origin
-   X-Varnish: 52136120
-   Age: 0
-   Via: 1.1 varnish (Varnish/6.5)
-   Strict-Transport-Security: max-age=15768000;
-   Connection: keep-alive
- Location: https://swhvaultstorage.blob.core.windows.net/contents-uncompressed/94f715535502200f7a4ede59a85d22dda9fc3510?se=2024-11-16T15%3A05%3A52Z&sp=r&sv=2024-11-04&sr=b&rscd=attachment%3B%20filename%3D%22swh_1_dir_6b700f4b287aee509adbc723d0303098188604f4.tar.gz%22&sig=9DEL7O0yEMhxz3Wgvo3QgJ6180mkDIjuV4rKoBCTRTc%3D [following]
- --2024-11-15 16:05:52--  https://swhvaultstorage.blob.core.windows.net/contents-uncompressed/94f715535502200f7a4ede59a85d22dda9fc3510?se=2024-11-16T15%3A05%3A52Z&sp=r&sv=2024-11-04&sr=b&rscd=attachment%3B%20filename%3D%22swh_1_dir_6b700f4b287aee509adbc723d0303098188604f4.tar.gz%22&sig=9DEL7O0yEMhxz3Wgvo3QgJ6180mkDIjuV4rKoBCTRTc%3D
- Resolving swhvaultstorage.blob.core.windows.net (swhvaultstorage.blob.core.windows.net)... 20.209.11.33
- Connecting to swhvaultstorage.blob.core.windows.net (swhvaultstorage.blob.core.windows.net)|20.209.11.33|:443... connected.
- HTTP request sent, awaiting response... 
-   HTTP/1.1 200 OK
-   Content-Length: 4213
-   Content-Type: application/octet-stream
-   Content-MD5: +jpEuytdGqLxCrKdtXLUMg==
-   Last-Modified: Wed, 19 Jun 2024 21:00:34 GMT
-   Accept-Ranges: bytes
-   ETag: "0x8DC90A2DD052E92"
-   Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
-   x-ms-request-id: 041ed48b-601e-0027-456f-37e913000000
-   x-ms-version: 2024-11-04
-   x-ms-creation-time: Wed, 19 Jun 2024 21:00:34 GMT
-   x-ms-lease-status: unlocked
-   x-ms-lease-state: available
-   x-ms-blob-type: BlockBlob
-   Content-Disposition: attachment; filename="swh_1_dir_6b700f4b287aee509adbc723d0303098188604f4.tar.gz"
-   x-ms-server-encrypted: true
-   Access-Control-Allow-Origin: *
-   Date: Fri, 15 Nov 2024 15:05:52 GMT
- Length: 4213 (4.1K) [application/octet-stream]
- Saving to: ‘/tmp/opam-1100962-3d90aa/6b700f4b287aee509adbc723d0303098188604f4.part’
- 
-      0K ....                                                  100%  268M=0s
- 
- 2024-11-15 16:05:52 (268 MB/s) - ‘/tmp/opam-1100962-3d90aa/6b700f4b287aee509adbc723d0303098188604f4.part’ saved [4213/4213]
- 
+ /usr/bin/tar "xfz" "/tmp/opam-1100962-3d90aa/6b700f4b287aee509adbc723d0303098188604f4" "-C" "/tmp/opam-1100962-c8b6c4"
⬇ retrieved swh-wget-cooked.1  (SWH fallback)
[...]

cURL & cooked

> ./opam install  -vvv --debug -y swh-curl-cooked
[...]
The following actions will be performed:
=== install 1 package
  ∗ swh-curl-cooked 1

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
Processing  1/3: [swh-curl-cooked.1: http]
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "/tmp/opam-1090321-caeea4/i-am-not-available-fake.o.r.g.part" "--" "https://i-am-not-available-fake.o.r.g/"
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: i-am-not-available-fake.o.r.g
- Warning: Problem : timeout. Will retry in 2 seconds. 3 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: i-am-not-available-fake.o.r.g
- Warning: Problem : timeout. Will retry in 2 seconds. 2 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: i-am-not-available-fake.o.r.g
- Warning: Problem : timeout. Will retry in 2 seconds. 1 retries left.
- 000
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: i-am-not-available-fake.o.r.g
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "-" "--" "https://archive.softwareheritage.org/api/1/ping/"
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
100     6  100     6    0     0     10      0 --:--:-- --:--:-- --:--:--    10
- "pong"200
00:21.231  CURL(SWHID)            API is working
Source https://i-am-not-available-fake.o.r.g/ is not available. Do you want to try to retrieve it from
Software Heritage cache (https://www.softwareheritage.org)? It may take few minutes. [y/n] y
00:21.231  CURL(SWHID)            SWH fallback for swh-curl-cooked.1 with swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4
Processing  1/3:
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "/opamroot/log/swh-out-1090321-b8392a" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4/"
- 200
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
100   241  100   241    0     0   1173      0 --:--:-- --:--:-- --:--:--  1169
00:21.955  CURL(SWHID)            swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4 is cooked, requesting url
00:21.956  CURL(SWHID)            Downloading swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4 for swh-curl-cooked.1
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.4.0~alpha1~dev" "-L" "-o" "/tmp/opam-1090321-fb6640/6b700f4b287aee509adbc723d0303098188604f4.part" "--" "https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:6b700f4b287aee509adbc723d0303098188604f4/raw/"
- 200
-   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
-                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  4213  100  4213    0     0   7233      0 --:--:-- --:--:-- --:--:--  7233
+ /usr/bin/tar "xfz" "/tmp/opam-1090321-fb6640/6b700f4b287aee509adbc723d0303098188604f4" "-C" "/tmp/opam-1090321-5ea6ea"
⬇ retrieved swh-curl-cooked.1  (SWH fallback)
[...]

Some logging enhancement were done after the output: counting of tries, and change swhid is cooked to swhid is cooked or cooking as it may be currently cooking.

kit-ty-kate

is the download: parse http error code for wget too and permit to retrieve it. commit absolutely necessary for this fix or can it be moved to a separate PR and be discussed separately? If this is possible i would rather review this change separately (with repository: add re as a dependency of opam-repository as both are related) so that this fix gets merged quicker (e.g. i personally don't think it's a good idea to parse --server-response)

kit-ty-kate · 2024-11-16T00:28:58Z

src/repository/opamDownload.ml

@@ -261,7 +261,7 @@ let get_output ~post ?(args=[]) url =
  let cmd_args =
    if post then
      match cmd_args with
-      | ("curl" as cmd)::args -> Some (cmd :: ["-X"; "POST"] @ args)
+      | ("curl" as cmd)::args -> Some (cmd :: ["--request"; "POST"] @ args)


what is the reasoning behind this change?

It's a personal view: i think it's better to use long form of options, it's easier to understand.

src/repository/opamDownload.ml

master_changes.md

rjbou · 2024-11-27T17:36:17Z

It'll be ported to another PR, like that if one day we want to have wget support, we have the code almost ready.

it's in #6314

kit-ty-kate

apart from the minor suggestion, lgtm. I'll review it more properly tomorrow but it looks very good. I especially like the change to a match in tool_return, it looks better i think.

src/repository/opamDownload.ml

kit-ty-kate

full review is still positive, i have a bit more suggestions though

master_changes.md

src/repository/opamDownload.ml

src/core/opamFilename.ml

rjbou · 2024-11-28T14:50:36Z

src/repository/opamDownload.ml

+    | Some out -> out
+    | None ->
+      (* Shouldn't happen, we already checked that a post tool is used *)
+      (* XXX change to an assert false ? *)


I don't think this display is needed, as in this module the main entry archive_fallback checks that.

in that case i agree this could be replaced by assert false

src/repository/opamDownload.ml

.gitignore

tests/reftests/swhid.unix.test

master_changes.md

…grained curl handling

…emporary files: 'with_tmp_file' and 'with_tmp_file_job'

* Update to new API * More fine-grained workflow: check is archive is already cooked: if not request cooking, otherwise retrieve url of the archive * Use only post request when needed

rjbou · 2024-12-03T17:45:02Z

Updated, and tested with cooked and uncooked packages

kit-ty-kate · 2024-12-03T19:03:15Z

Thanks!

rjbou added this to the 2.2.0~rc1 milestone Jun 19, 2024

rjbou requested a review from kit-ty-kate June 19, 2024 17:47

rjbou linked an issue Jun 19, 2024 that may be closed by this pull request

opam does not wait for the Software Heritage fallback archives to be "cooked" #5721

Closed

rjbou added the KIND: BUG label Jun 19, 2024

dra27 reviewed Jun 19, 2024

View reviewed changes

src/repository/opamDownload.ml Outdated Show resolved Hide resolved

src/repository/opamDownload.ml Outdated Show resolved Hide resolved

src/repository/opamDownload.ml Outdated Show resolved Hide resolved

src/repository/opamDownload.ml Outdated Show resolved Hide resolved

rjbou force-pushed the swhid-test branch from 1442734 to 1e2f479 Compare June 19, 2024 21:12

dra27 approved these changes Jun 20, 2024

View reviewed changes

rjbou added the PR: NEEDS UPDATE label Jun 20, 2024

rjbou removed this from the 2.2.0~rc1 milestone Jun 21, 2024

rjbou added this to the 2.3 milestone Jun 27, 2024

kit-ty-kate approved these changes Jul 9, 2024

View reviewed changes

rjbou marked this pull request as draft July 10, 2024 14:15

kit-ty-kate modified the milestones: 2.3.0~alpha1, 2.4.0~alpha1 Sep 18, 2024

rjbou force-pushed the swhid-test branch from 1e2f479 to 40fe666 Compare November 15, 2024 16:58

rjbou added PR: WAITING FOR REVIEW and removed PR: NEEDS UPDATE labels Nov 15, 2024

rjbou force-pushed the swhid-test branch from 40fe666 to 34a5ec0 Compare November 15, 2024 17:06

rjbou requested review from dra27 and kit-ty-kate November 15, 2024 17:07

rjbou marked this pull request as ready for review November 15, 2024 17:07

kit-ty-kate force-pushed the swhid-test branch from 34a5ec0 to 9c7635b Compare November 16, 2024 00:27

kit-ty-kate reviewed Nov 16, 2024

View reviewed changes

rjbou added PR: NEEDS UPDATE and removed PR: WAITING FOR REVIEW labels Nov 18, 2024

rjbou commented Nov 27, 2024

View reviewed changes

master_changes.md Outdated Show resolved Hide resolved

rjbou force-pushed the swhid-test branch from 99a9d6c to 349dba9 Compare November 27, 2024 17:31

kit-ty-kate reviewed Nov 27, 2024

View reviewed changes

src/repository/opamDownload.ml Outdated Show resolved Hide resolved

kit-ty-kate reviewed Nov 27, 2024

View reviewed changes

rjbou force-pushed the swhid-test branch from 349dba9 to 433c92b Compare November 28, 2024 14:48

rjbou commented Nov 28, 2024

View reviewed changes

src/core/opamFilename.ml Show resolved Hide resolved

rjbou commented Nov 28, 2024

View reviewed changes

src/repository/opamDownload.ml Outdated Show resolved Hide resolved

kit-ty-kate reviewed Nov 28, 2024

View reviewed changes

.gitignore Outdated Show resolved Hide resolved

rjbou commented Nov 28, 2024

View reviewed changes

tests/reftests/swhid.unix.test Outdated Show resolved Hide resolved

rjbou commented Nov 28, 2024

View reviewed changes

master_changes.md Show resolved Hide resolved

rjbou added 11 commits December 3, 2024 18:43

gitignore: add release binary output directory

eafd1f5

download: fix POST option for wget

55306c1

download: use long format option for curl POST request

f703625

download: remove wget support for SWH fallback to focus on more fine-…

70c0a40

…grained curl handling

download: fix SWH liveness check

cde8604

download: fix SWH cooking request and status retrieval API request

cadcf75

download: add internal function to retrieve http code for curl.

b1926cf

core: factorise temporary file and directory names creation

7fd8377

core: add 'OpamSystem' and 'OpamFilename' function to use and clean t…

4df6e02

…emporary files: 'with_tmp_file' and 'with_tmp_file_job'

download: rework SWH retrieval fallback

7ef8a80

* Update to new API * More fine-grained workflow: check is archive is already cooked: if not request cooking, otherwise retrieve url of the archive * Use only post request when needed

download: add some debug logging for SWH fallback

530fd27

rjbou force-pushed the swhid-test branch from a61576f to 530fd27 Compare December 3, 2024 17:44

rjbou removed the PR: NEEDS UPDATE label Dec 3, 2024

kit-ty-kate approved these changes Dec 3, 2024

View reviewed changes

rjbou added the STATE: READY TO MERGE label Dec 3, 2024

kit-ty-kate merged commit efcfa64 into ocaml:master Dec 3, 2024
42 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Software Heritage fallback fixes #6036

Software Heritage fallback fixes #6036

rjbou commented Jun 19, 2024 •

edited

Loading

dra27 left a comment

dra27 left a comment

rjbou commented Nov 15, 2024

kit-ty-kate left a comment

kit-ty-kate Nov 16, 2024

rjbou Nov 18, 2024

rjbou commented Nov 27, 2024

kit-ty-kate left a comment

kit-ty-kate left a comment

rjbou Nov 28, 2024

kit-ty-kate Nov 28, 2024

rjbou commented Dec 3, 2024

kit-ty-kate commented Dec 3, 2024

Software Heritage fallback fixes #6036

Software Heritage fallback fixes #6036

Conversation

rjbou commented Jun 19, 2024 • edited Loading

dra27 left a comment

Choose a reason for hiding this comment

dra27 left a comment

Choose a reason for hiding this comment

rjbou commented Nov 15, 2024

kit-ty-kate left a comment

Choose a reason for hiding this comment

kit-ty-kate Nov 16, 2024

Choose a reason for hiding this comment

rjbou Nov 18, 2024

Choose a reason for hiding this comment

rjbou commented Nov 27, 2024

kit-ty-kate left a comment

Choose a reason for hiding this comment

kit-ty-kate left a comment

Choose a reason for hiding this comment

rjbou Nov 28, 2024

Choose a reason for hiding this comment

kit-ty-kate Nov 28, 2024

Choose a reason for hiding this comment

rjbou commented Dec 3, 2024

kit-ty-kate commented Dec 3, 2024

rjbou commented Jun 19, 2024 •

edited

Loading