Relax nanosecond datetime restriction in CF time decoding

[kmuehlbauer]

• Closes Interoperability with Pandas 2.0 non-nanosecond datetime #7493
• Closes DataArray constructor still coerces to np.datetime64[ns], not cftime in 0.11.0 #2587
• Tests added/changed
• User visible changes (including notable bug fixes) are documented in whats-new.rst
• New functions/methods are listed in api.rst

This is another attempt to resolve #7493. This goes a step further than #9580.

The idea of this PR is to automatically infer the needed resolutions for decoding/encoding and only keep the constraints pandas imposes ("s" - lowest resolution, "ns" - highest resolution). There is still the idea of a default resolution, but this should only take precedence if it doesn't clash with the automatic inference. This can be discussed, though. Update: I've implemented time-unit-kwarg a first try to have default resolution on decode, which will override the current inferred resolution only to higher resolution (eg. 's' -> 'ns'). To work towards #4490 the time decoding options (decode_time and use_cftime are bundled within CFDatetimeCoder which is distributed via decode_times kwarg. use_cftime-kwarg is deprecated.

For sanity checking, and also for my own good, I've created a documentation page on time-coding in the internal dev section. Any suggestions (especially grammar) or ideas for enhancements are much appreciated.

There still might be room for consolidation of functions/methods (mostly in coding/times.py), but I have to leave it alone for some days. I went down that rabbit hole and need to relax, too 😬.

Looking forward to get your insights here, @spencerkclark, @ChrisBarker-NOAA, @pydata/xarray.

Todo:

• floating point handling
• update decoding tests to iterate over time_units (where appropriate)
• CFDatetimeCoder as input for decode_times kwarg
• ...

[kmuehlbauer]

Nice, mypy 1.12 is out and breaks our typing, 😭.

[TomNicholas]

Nice, mypy 1.12 is out and breaks our typing, 😭

Can we pin it in the CI temporarily?

[kmuehlbauer]

Can we pin it in the CI temporarily?

Yes, 1.11.2 was the last version.

[kmuehlbauer]

This is now ready for a first round of review. I think this is already in a quite usable state.

But no rush, this should be thoroughly tested.

[spencerkclark]

Sounds good @kmuehlbauer! I’ll try and take an initial look this weekend.

[alippai]

I see many notes about the units s-us. Will this work with minutes or days too? Eg date_range and other sources

[kmuehlbauer]

Thank you for the review Deepak, ❤️

I'll try to get along over the weekend.

[kmuehlbauer]

Todo: Fix version

[kmuehlbauer]

TODO: Fix version

[kmuehlbauer]

I'll look into it.

[kmuehlbauer]

I see many notes about the units s-us. Will this work with minutes or days too? Eg date_range and other sources

@alippai Can you give an example what you have in mind? Currently xarray holds datetimes as np.datetime64[ns] under the hood. This PR relaxes this restriction so datetimes can be represented as np.datetime64[s], np.datetime64[ms] and np.datetime64[us], too.

[kmuehlbauer]

We need to have a big note in whats-new.rst about behaviour changes.

1. The default for `pd.date_range` seems to have changed, and we now preserve the unit instead of casting. This will impact downstream code.

2. Has the default unit for `xr.date_range` changed? We'll need to add the `unit` kwarg with default `ns` and start warning about switching to `us` by default.

3. What is the impact on `polyfit` and `differentiate`? These are places where I (and many others) have manually rescaled from `ns` to other units dividing by `1e9`. We don't want to break this silently.

Thanks @dcherian for bringing this to attention. I think we can circumvent these behaviour changes. Let me try first.

[alippai]

I see many notes about the units s-us. Will this work with minutes or days too? Eg date_range and other sources

@alippai Can you give an example what you have in mind? Currently xarray holds datetimes as np.datetime64[ns] under the hood. This PR relaxes this restriction so datetimes can be represented as np.datetime64[s], np.datetime64[ms] and np.datetime64[us], too.

Larger units like np.datetime64[D] and m types

[kmuehlbauer]

@dcherian @spencerkclark I'm now trying to split out everything which helps to get the footprint lower here:

• finalize deprecation of "closed"-parameter #9882
• Add "unit"-parameter to date_range, enhance iso time parser to us #9885
• Enhance and move ISO-8601 parser to coding.times #9899
• use iso-parser when reference time out-of-bounds (needs Enhance and move ISO-8601 parser to coding.times #9899 and time coding refactor #9906)
• move scalar-handling logic into possibly_convert_objects #9900
• split out CFDatetimeCoder, deprecate use_cftime as kwarg #9901
• time coding refactor #9906
• tbc