[Question] Unable to retrieve SDKs from oryx-cdn with VNet integration

[tanyunshi]

Hello,

We need helps to troubleshooting the "Unable to retrieve SDKs from oryx-cdn" issue while deploying the Azure Function with a VNet and in a app service plan tier B3.

• We are aware the network dependency to oryx-cdn.microsoft.io as mentioned in the doc.

• https://oryx-cdn.microsoft.io is well whitelisted by the firewall.
  In the host ( via KUDU UI/SSH ), CURL https://oryx-cdn.microsoft.io tells that the oryx cdn is accessible.

root@7cdde5d43bd2:/# curl https://oryx-cdn.microsoft.io
<?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidQueryParameterValue</Code><Message>Value for one of the query parameters specified in the request URI is invalid.
RequestId:3c99b046-701e-0090-73aa-e43233000000
Time:2024-08-02T07:04:53.6667287Z</Message><QueryParameterName>comp</QueryParameterName><QueryParameterValue /><Reason /></Error>

• The function app is in an app service plan B3. We tried P3 but the issue persists.

• There is no Network security group configured.

• There is no NAT gateway.

• There is an user defined route. All traffic is sent forward to the firewall. The Oryx CDN is well whitelisted by the firewall ( since curl ok)

• The build error

Command: oryx build /tmp/zipdeploy/extracted -o /tmp/build/expressbuild --platform python --platform-version 3.11 -i /tmp/8dcb251b8c71745 -p packagedir=.python_packages/lib/site-packages
...
Oryx Version: 0.2.20240424.1, Commit: d37b2225a252ab2c04b4726024d047cf01ea1318, ReleaseTagName: 20240424.1
...
Error: System.AggregateException: Http request to retrieve the SDKs available to download from 'https://oryx-cdn.microsoft.io' failed. Please ensure that your network configuration allows traffic to required Oryx dependencies, as documented in 'https://github.com/microsoft/Oryx/blob/main/doc/hosts/appservice.md#network-dependencies
System.AggregateException: One or more errors occurred. (A task was canceled.)
...

I do not know how can we debug the issue ? I will be grateful if you can give us some advice :)

[sarsharma]

Hi @tanyunshi,
can you please try to run that curl command via Kudu bash
The curl logs you attached look like the request was made via webssh

This difference is important since the oryx build happens inside the Kudu container.
can you run this curl request in Kudu bash curl -O https://oryx-cdn.microsoft.io/python/python-3.12.1.tar.gz and share if that works.
Thanks!

[tanyunshi]

Hello @sarsharma

Thanks for your response !

Indeed the command curl -O https://oryx-cdn.microsoft.io/python/python-3.12.1.tar.gz runs well via webssh.

Via Kudu Bash, the curl fails with an certificate error unable to get local issuer certificate

kudu_ssh_user@b4885cbc9e2e:/$ curl -O https://oryx-cdn.microsoft.io/python/python-3.12.1.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

With the option "curl -k", I can down load the package via Kudu Bash.

Shall I install some certificates for the host ?

[vtancardoso]

@tanyunshi were you able to find a workaround? We are having the same issues as well

[tanyunshi]

Hello @vtancardoso

I am still looking for a workaround.

In my case, the firewall of the company inspect the TLS communication. It means that the traffic is decrypted and then re-encrypted. So the certificate is replaced by a custom one, generated on the fly at the company's side.

The custom's CA is uploaded via Function App. That's why in the web ssh console (the application's container), the CURL works well.

In the KUDU container, the CURL fails with the certificate issue (as suggested by @sarsharma). And indeed in the /etc/ssl/certs in the KUDU bash, the custom's CA is not there.

So I guess, either we whitelist the TLS inspection ( which involves the network team), either we find a way to upload the custom CA in the KUDU container ( and I doubt if it is possible .... )

[tanyunshi]

Hello,

Finally we give up using Oryx because changing the network configuration requires lots of administration steps...

We build the app in local and push the package to the Func App as describe here.

Thank you again for your help. and I am closing the issue.