Invalid LocationConstraint causing error on non-aws (ceph) object storage services

[zliang-akamai]

Summary

The ansible module relies on AWS's boto3 library to determine the default region for s3 bucket creation:

amazon.aws/plugins/module_utils/botocore.py

Line 224 in da0027d

return boto3.session.Session(profile_name=profile_name).region_name

Many third party object storage service utilizing Ceph clusters doesn't accept AWS regions and requires LocationConstraint to be a specific value of being omitted from the request.

https://docs.ceph.com/en/latest/radosgw/placement/#s3-bucket-placement

So when ceph attribute is set to true and endpoint_url is provided to the amazon.aws.s3_bucket module, I think it shouldn't let AWS's boto3 to automatically detect an AWS region and set that AWS region into LocationConstraint, whcih can cause error InvalidLocationConstraint returning from Ceph clusters.

Issue Type

Bug Report

Component Name

s3_bucket

Ansible Version

$ ansible --version
ansible [core 2.18.1]
  config file = None
  configured module search path = ['/Users/myusername/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/myusername/path/to/ansible_collections/linode/cloud/venv/lib/python3.11/site-packages/ansible
  ansible collection location = /Users/myusername/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/myusername/path/to/ansible_collections/linode/cloud/venv/bin/ansible
  python version = 3.11.9 (v3.11.9:de54cf5be3, Apr  2 2024, 07:12:50) [Clang 13.0.0 (clang-1300.0.29.30)] (/Users/myusername/path/to/ansible_collections/linode/cloud/venv/bin/python3.11)
  jinja version = 3.1.4
  libyaml = True

Collection Versions

$ ansible-galaxy collection list
# /Users/myusername/.ansible/collections/ansible_collections
Collection                               Version    
---------------------------------------- -----------
amazon.aws                               10.0.0-dev0
linode.cloud                             0.0.0      

# /Users/myusername/path/to/ansible_collections/linode/cloud/venv/lib/python3.11/site-packages/ansible_collections
Collection                               Version    
---------------------------------------- -----------
amazon.aws                               9.0.0      
ansible.netcommon                        7.1.0      
ansible.posix                            1.6.2      
ansible.utils                            5.1.2      
ansible.windows                          2.5.0      
arista.eos                               10.0.1     
awx.awx                                  24.6.1     
azure.azcollection                       3.1.0      
check_point.mgmt                         6.2.1      
chocolatey.chocolatey                    1.5.3      
cisco.aci                                2.10.1     
cisco.asa                                6.0.0      
cisco.dnac                               6.25.0     
cisco.intersight                         2.0.20     
cisco.ios                                9.0.3      
cisco.iosxr                              10.2.2     
cisco.ise                                2.9.6      
cisco.meraki                             2.18.3     
cisco.mso                                2.9.0      
cisco.nxos                               9.2.1      
cisco.ucs                                1.14.0     
cloud.common                             4.0.0      
cloudscale_ch.cloud                      2.4.0      
community.aws                            9.0.0      
community.ciscosmb                       1.0.9      
community.crypto                         2.22.3     
community.digitalocean                   1.27.0     
community.dns                            3.1.0      
community.docker                         4.1.0      
community.general                        10.1.0     
community.grafana                        2.1.0      
community.hashi_vault                    6.2.0      
community.hrobot                         2.0.2      
community.library_inventory_filtering_v1 1.0.2      
community.libvirt                        1.3.0      
community.mongodb                        1.7.8      
community.mysql                          3.11.0     
community.network                        5.1.0      
community.okd                            4.0.0      
community.postgresql                     3.9.0      
community.proxysql                       1.6.0      
community.rabbitmq                       1.3.0      
community.routeros                       3.1.0      
community.sap_libs                       1.4.2      
community.sops                           2.0.0      
community.vmware                         5.2.0      
community.windows                        2.3.0      
community.zabbix                         3.2.0      
containers.podman                        1.16.2     
cyberark.conjur                          1.3.1      
cyberark.pas                             1.0.30     
dellemc.enterprise_sonic                 2.5.1      
dellemc.openmanage                       9.9.0      
dellemc.powerflex                        2.5.0      
dellemc.unity                            2.0.0      
f5networks.f5_modules                    1.32.1     
fortinet.fortimanager                    2.8.2      
fortinet.fortios                         2.3.8      
google.cloud                             1.4.1      
grafana.grafana                          5.6.0      
hetzner.hcloud                           4.2.2      
ibm.qradar                               4.0.0      
ibm.spectrum_virtualize                  2.0.0      
ibm.storage_virtualize                   2.5.0      
ieisystem.inmanage                       3.0.0      
infinidat.infinibox                      1.4.5      
infoblox.nios_modules                    1.7.1      
inspur.ispim                             2.2.3      
junipernetworks.junos                    9.1.0      
kaytus.ksmanage                          2.0.0      
kubernetes.core                          5.0.0      
kubevirt.core                            2.1.0      
lowlydba.sqlserver                       2.3.4      
microsoft.ad                             1.7.1      
netapp.cloudmanager                      21.24.0    
netapp.ontap                             22.13.0    
netapp.storagegrid                       21.13.0    
netapp_eseries.santricity                1.4.1      
netbox.netbox                            3.20.0     
ngine_io.cloudstack                      2.5.0      
openstack.cloud                          2.3.0      
ovirt.ovirt                              3.2.0      
purestorage.flasharray                   1.32.0     
purestorage.flashblade                   1.19.1     
sensu.sensu_go                           1.14.0     
splunk.es                                4.0.0      
telekom_mms.icinga_director              2.2.1      
theforeman.foreman                       4.2.0      
vmware.vmware                            1.7.1      
vmware.vmware_rest                       4.3.0      
vultr.cloud                              1.13.0     
vyos.vyos                                5.0.0      
wti.remote                               1.0.10

AWS SDK versions

$ pip show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.35.77
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /Users/myusername/path/to/ansible_collections/linode/cloud/venv/lib/python3.11/site-packages
Requires: botocore, jmespath, s3transfer
Required-by: 
---
Name: botocore
Version: 1.35.77
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /Users/myusername/path/to/ansible_collections/linode/cloud/venv/lib/python3.11/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

$ ansible-config dump --only-changed

OS / Environment

macOS 14

Steps to Reproduce

- name: Create an S3 bucket
  amazon.aws.s3_bucket:
    ceph: true
    endpoint_url: 'https://{{ myendpoint.endpoint }}/'
    aws_access_key: '{{ mykey.access_key }}'
    aws_secret_key: '{{ mykey.secret_key }}'
    name: 'my-bucket'
    state: present
  register: create_bucket

Expected Results

Bucket created, LocationConstraint is not included in the request sent to Ceph S3 API.

Actual Results

Failed while creating bucket: An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation: The specified location-constraint is not valid

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[zliang-akamai]

If you can confirm this bug, I am happy to put up a PR for it