<a id="groupby"></a>

# GroupBy: Group and Bin Data

Often we want to bin or group data, produce statistics (mean, variance) on
the groups, and then return a reduced data set. To do this, Xarray supports
[“group by”](https://pandas.pydata.org/pandas-docs/stable/groupby.html) operations with the same API as pandas to implement the
[split-apply-combine](https://www.jstatsoft.org/v40/i01/paper) strategy:

- Split your data into multiple independent groups.
- Apply some function to each group.
- Combine your groups back into a single data object.

Group by operations work on both [`Dataset`](../generated/xarray.Dataset.html.md#xarray.Dataset) and
[`DataArray`](../generated/xarray.DataArray.html.md#xarray.DataArray) objects. Most of the examples focus on grouping by
a single one-dimensional variable, although support for grouping
over a multi-dimensional variable has recently been implemented. Note that for
one-dimensional data, it is usually faster to rely on pandas’ implementation of
the same pipeline.

#### TIP
[Install the flox package](https://flox.readthedocs.io) to substantially improve the performance
of GroupBy operations, particularly with dask. flox
[extends Xarray’s in-built GroupBy capabilities](https://flox.readthedocs.io/en/latest/xarray.html)
by allowing grouping by multiple variables, and lazy grouping by dask arrays.
If installed, Xarray will automatically use flox by default.

## Split

Let’s create a simple example dataset:

```ipython3
ds = xr.Dataset(
    {"foo": (("x", "y"), np.random.rand(4, 3))},
    coords={"x": [10, 20, 30, 40], "letters": ("x", list("abba"))},
)
arr = ds["foo"]
ds
```

If we groupby the name of a variable or coordinate in a dataset (we can also
use a DataArray directly), we get back a `GroupBy` object:

```ipython3
ds.groupby("letters")
```

This object works very similarly to a pandas GroupBy object. You can view
the group indices with the `groups` attribute:

```ipython3
ds.groupby("letters").groups
```

You can also iterate over groups in `(label, group)` pairs:

```ipython3
list(ds.groupby("letters"))
```

You can index out a particular group:

```ipython3
ds.groupby("letters")["b"]
```

To group by multiple variables, see [this section](#groupby-multiple).

## Binning

Sometimes you don’t want to use all the unique values to determine the groups
but instead want to “bin” the data into coarser groups. You could always create
a customized coordinate, but xarray facilitates this via the
[`Dataset.groupby_bins()`](../generated/xarray.Dataset.groupby_bins.html.md#xarray.Dataset.groupby_bins) method.

```ipython3
x_bins = [0, 25, 50]
ds.groupby_bins("x", x_bins).groups
```

The binning is implemented via [`pandas.cut()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.cut.html#pandas.cut), whose documentation details how
the bins are assigned. As seen in the example above, by default, the bins are
labeled with strings using set notation to precisely identify the bin limits. To
override this behavior, you can specify the bin labels explicitly. Here we
choose `float` labels which identify the bin centers:

```ipython3
x_bin_labels = [12.5, 37.5]
ds.groupby_bins("x", x_bins, labels=x_bin_labels).groups
```

## Apply

To apply a function to each group, you can use the flexible
[`core.groupby.DatasetGroupBy.map()`](../generated/xarray.core.groupby.DatasetGroupBy.map.html.md#xarray.core.groupby.DatasetGroupBy.map) method. The resulting objects are automatically
concatenated back together along the group axis:

```ipython3
def standardize(x):
    return (x - x.mean()) / x.std()


arr.groupby("letters").map(standardize)
```

GroupBy objects also have a [`core.groupby.DatasetGroupBy.reduce()`](../generated/xarray.core.groupby.DatasetGroupBy.reduce.html.md#xarray.core.groupby.DatasetGroupBy.reduce) method and
methods like [`core.groupby.DatasetGroupBy.mean()`](../generated/xarray.core.groupby.DatasetGroupBy.mean.html.md#xarray.core.groupby.DatasetGroupBy.mean) as shortcuts for applying an
aggregation function:

```ipython3
arr.groupby("letters").mean(dim="x")
```

Using a groupby is thus also a convenient shortcut for aggregating over all
dimensions *other than* the provided one:

```ipython3
ds.groupby("x").std(...)
```

#### NOTE
We use an ellipsis (`...`) here to indicate we want to reduce over all
other dimensions

## First and last

There are two special aggregation operations that are currently only found on
groupby objects: first and last. These provide the first or last example of
values for group along the grouped dimension:

```ipython3
ds.groupby("letters").first(...)
```

By default, they skip missing values (control this with `skipna`).

## Grouped arithmetic

GroupBy objects also support a limited set of binary arithmetic operations, as
a shortcut for mapping over all unique labels. Binary arithmetic is supported
for `(GroupBy, Dataset)` and `(GroupBy, DataArray)` pairs, as long as the
dataset or data array uses the unique grouped values as one of its index
coordinates. For example:

```ipython3
alt = arr.groupby("letters").mean(...)
alt
```

```ipython3
ds.groupby("letters") - alt
```

This last line is roughly equivalent to the following:

```default
results = []
for label, group in ds.groupby('letters'):
    results.append(group - alt.sel(letters=label))
xr.concat(results, dim='x')
```

<a id="groupby-multidim"></a>

## Multidimensional Grouping

Many datasets have a multidimensional coordinate variable (e.g. longitude)
which is different from the logical grid dimensions (e.g. nx, ny). Such
variables are valid under the [CF conventions](https://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html#_two_dimensional_latitude_longitude_coordinate_variables). Xarray supports groupby
operations over multidimensional coordinate variables:

```ipython3
da = xr.DataArray(
    [[0, 1], [2, 3]],
    coords={
        "lon": (["ny", "nx"], [[30, 40], [40, 50]]),
        "lat": (["ny", "nx"], [[10, 10], [20, 20]]),
    },
    dims=["ny", "nx"],
)
da
```

```ipython3
da.groupby("lon").sum(...)
```

```ipython3
da.groupby("lon").map(lambda x: x - x.mean(), shortcut=False)
```

Because multidimensional groups have the ability to generate a very large
number of bins, coarse-binning via [`Dataset.groupby_bins()`](../generated/xarray.Dataset.groupby_bins.html.md#xarray.Dataset.groupby_bins)
may be desirable:

```ipython3
da.groupby_bins("lon", [0, 45, 50]).sum()
```

These methods group by `lon` values. It is also possible to groupby each
cell in a grid, regardless of value, by stacking multiple dimensions,
applying your function, and then unstacking the result:

```ipython3
stacked = da.stack(gridcell=["ny", "nx"])
stacked.groupby("gridcell").sum(...).unstack("gridcell")
```

Alternatively, you can groupby both `lat` and `lon` at the [same time](#groupby-multiple).

<a id="groupby-groupers"></a>

## Grouper Objects

Both `groupby_bins` and `resample` are specializations of the core `groupby` operation for binning,
and time resampling. Many problems demand more complex GroupBy application: for example, grouping by multiple
variables with a combination of categorical grouping, binning, and resampling; or more specializations like
spatial resampling; or more complex time grouping like special handling of seasons, or the ability to specify
custom seasons. To handle these use-cases and more, Xarray is evolving to providing an
extension point using `Grouper` objects.

#### TIP
See the [grouper design](https://github.com/pydata/xarray/blob/main/design_notes/grouper_objects.md) doc for more detail on the motivation and design ideas behind
Grouper objects.

For now Xarray provides three specialized Grouper objects:

1. [`groupers.UniqueGrouper`](../generated/xarray.groupers.UniqueGrouper.html.md#xarray.groupers.UniqueGrouper) for categorical grouping
2. [`groupers.BinGrouper`](../generated/xarray.groupers.BinGrouper.html.md#xarray.groupers.BinGrouper) for binned grouping
3. [`groupers.TimeResampler`](../generated/xarray.groupers.TimeResampler.html.md#xarray.groupers.TimeResampler) for resampling along a datetime coordinate

These provide functionality identical to the existing `groupby`, `groupby_bins`, and `resample` methods.
That is,

```python
ds.groupby("x")
```

is identical to

```python
from xarray.groupers import UniqueGrouper

ds.groupby(x=UniqueGrouper())
```

Similarly,

```python
ds.groupby_bins("x", bins=bins)
```

is identical to

```python
from xarray.groupers import BinGrouper

ds.groupby(x=BinGrouper(bins))
```

and

```python
ds.resample(time="ME")
```

is identical to

```python
from xarray.groupers import TimeResampler

ds.resample(time=TimeResampler("ME"))
```

The [`groupers.UniqueGrouper`](../generated/xarray.groupers.UniqueGrouper.html.md#xarray.groupers.UniqueGrouper) accepts an optional `labels` kwarg that is not present
in [`DataArray.groupby()`](../generated/xarray.DataArray.groupby.html.md#xarray.DataArray.groupby) or [`Dataset.groupby()`](../generated/xarray.Dataset.groupby.html.md#xarray.Dataset.groupby).
Specifying `labels` is required when grouping by a lazy array type (e.g. dask or cubed).
The `labels` are used to construct the output coordinate (say for a reduction), and aggregations
will only be run over the specified labels.
You may use `labels` to also specify the ordering of groups to be used during iteration.
The order will be preserved in the output.

<a id="groupby-multiple"></a>

## Grouping by multiple variables

Use grouper objects to group by multiple dimensions:

```ipython3
from xarray.groupers import UniqueGrouper

da.groupby(["lat", "lon"]).sum()
```

When grouping by multiple variables, combinations that are not present in the
data may also be included. To group only by observed combinations, you can
combine the variables into a `PandasMultiIndex`:

```ipython3
import numpy as np
import xarray as xr
from xarray.indexes import PandasMultiIndex

xdata = xr.DataArray(
    np.ones((4, 2)),
    coords={
        "y": [0, 1, 2, 3],
        "y1": ("y", [1, 1, 2, 2]),
        "y2": ("y", [1, 1, 2, 2]),
    },
    dims=["y", "x"],
)

xdata = xdata.set_xindex(["y1", "y2"], PandasMultiIndex)
xdata.groupby("y").mean()
```

The above is sugar for using `UniqueGrouper` objects directly:

```ipython3
da.groupby(lat=UniqueGrouper(), lon=UniqueGrouper()).sum()
```

Different groupers can be combined to construct sophisticated GroupBy operations.

```ipython3
from xarray.groupers import BinGrouper

ds.groupby(x=BinGrouper(bins=[5, 15, 25]), letters=UniqueGrouper()).sum()
```

## Time Grouping and Resampling

#### SEE ALSO
See [Resampling and grouped operations](time-series.html.md#resampling).

## Shuffling

Shuffling is a generalization of sorting a DataArray or Dataset by another DataArray, named `label` for example, that follows from the idea of grouping by `label`.
Shuffling reorders the DataArray or the DataArrays in a Dataset such that all members of a group occur sequentially. For example,
Shuffle the object using either `DatasetGroupBy` or `DataArrayGroupBy` as appropriate.

```ipython3
da = xr.DataArray(
    dims="x",
    data=[1, 2, 3, 4, 5, 6],
    coords={"label": ("x", "a b c a b c".split(" "))},
)
da.groupby("label").shuffle_to_chunks()
```

For chunked array types (e.g. dask or cubed), shuffle may result in a more optimized communication pattern when compared to direct indexing by the appropriate indexer.
Shuffling also makes GroupBy operations on chunked arrays an embarrassingly parallel problem, and may significantly improve workloads that use `DatasetGroupBy.map()` or `DataArrayGroupBy.map()`.