<a id="pandas"></a>

# Working with pandas

One of the most important features of xarray is the ability to convert to and
from [`pandas`](https://pandas.pydata.org/pandas-docs/stable/index.html#module-pandas) objects to interact with the rest of the PyData
ecosystem. For example, for plotting labeled data, we highly recommend
using the [visualization built in to pandas itself](https://pandas.pydata.org/pandas-docs/stable/visualization.html) or provided by the pandas
aware libraries such as [Seaborn](https://seaborn.pydata.org/).

## Hierarchical and tidy data

Tabular data is easiest to work with when it meets the criteria for
[tidy data](https://www.jstatsoft.org/v59/i10/):

- Each column holds a different variable.
- Each rows holds a different observation.

In this “tidy data” format, we can represent any [`Dataset`](../generated/xarray.Dataset.html.md#xarray.Dataset) and
[`DataArray`](../generated/xarray.DataArray.html.md#xarray.DataArray) in terms of [`DataFrame`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html#pandas.DataFrame) and
[`Series`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.html#pandas.Series), respectively (and vice-versa). The representation
works by flattening non-coordinates to 1D, and turning the tensor product of
coordinate indexes into a [`pandas.MultiIndex`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.MultiIndex.html#pandas.MultiIndex).

### Dataset and DataFrame

To convert any dataset to a `DataFrame` in tidy form, use the
[`Dataset.to_dataframe()`](../generated/xarray.Dataset.to_dataframe.html.md#xarray.Dataset.to_dataframe) method:

```ipython3
ds = xr.Dataset(
    {"foo": (("x", "y"), np.random.randn(2, 3))},
    coords={
        "x": [10, 20],
        "y": ["a", "b", "c"],
        "along_x": ("x", np.random.randn(2)),
        "scalar": 123,
    },
)
ds
```

```ipython3
df = ds.to_dataframe()
df
```

We see that each variable and coordinate in the Dataset is now a column in the
DataFrame, with the exception of indexes which are in the index.
To convert the `DataFrame` to any other convenient representation,
use `DataFrame` methods like [`reset_index()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.reset_index.html#pandas.DataFrame.reset_index),
[`stack()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.stack.html#pandas.DataFrame.stack) and [`unstack()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.unstack.html#pandas.DataFrame.unstack).

For datasets containing dask arrays where the data should be lazily loaded, see the
[`Dataset.to_dask_dataframe()`](../generated/xarray.Dataset.to_dask_dataframe.html.md#xarray.Dataset.to_dask_dataframe) method.

To create a `Dataset` from a `DataFrame`, use the
[`Dataset.from_dataframe()`](../generated/xarray.Dataset.from_dataframe.html.md#xarray.Dataset.from_dataframe) class method or the equivalent
[`pandas.DataFrame.to_xarray()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_xarray.html#pandas.DataFrame.to_xarray) method:

```ipython3
xr.Dataset.from_dataframe(df)
```

Notice that the dimensions of variables in the `Dataset` have now
expanded after the round-trip conversion to a `DataFrame`. This is because
every object in a `DataFrame` must have the same indices, so we need to
broadcast the data of each array to the full size of the new `MultiIndex`.

Likewise, all the coordinates (other than indexes) ended up as variables,
because pandas does not distinguish non-index coordinates.

### DataArray and Series

`DataArray` objects have a complementary representation in terms of a
[`Series`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.html#pandas.Series). Using a Series preserves the `Dataset` to
`DataArray` relationship, because `DataFrames` are dict-like containers
of `Series`. The methods are very similar to those for working with
DataFrames:

```ipython3
s = ds["foo"].to_series()
s
```

```ipython3
# or equivalently, with Series.to_xarray()
xr.DataArray.from_series(s)
```

Both the `from_series` and `from_dataframe` methods use reindexing, so they
work even if the hierarchical index is not a full tensor product:

```ipython3
s[::2]
```

```ipython3
s[::2].to_xarray()
```

## Lossless and reversible conversion

The previous `Dataset` example shows that the conversion is not reversible (lossy roundtrip) and
that the size of the `Dataset` increases.

Particularly after a roundtrip, the following deviations are noted:

- a non-dimension Dataset `coordinate` is converted into `variable`
- a non-dimension DataArray `coordinate` is not converted
- `dtype` is not always the same (e.g. “str” is converted to “object”)
- `attrs` metadata is not conserved

To avoid these problems, the third-party [ntv-pandas](https://github.com/loco-philippe/ntv-pandas) library offers lossless and reversible conversions between
`Dataset`/ `DataArray` and pandas `DataFrame` objects.

This solution is particularly interesting for converting any `DataFrame` into a `Dataset` (the converter finds the multidimensional structure hidden by the tabular structure).

The [ntv-pandas examples](https://github.com/loco-philippe/ntv-pandas/tree/main/example) show how to improve the conversion for the previous `Dataset` example and for more complex examples.

## Multi-dimensional data

Tidy data is great, but it sometimes you want to preserve dimensions instead of
automatically stacking them into a `MultiIndex`.

[`DataArray.to_pandas()`](../generated/xarray.DataArray.to_pandas.html.md#xarray.DataArray.to_pandas) is a shortcut that lets you convert a
DataArray directly into a pandas object with the same dimensionality, if
available in pandas (i.e., a 1D array is converted to a
[`Series`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.html#pandas.Series) and 2D to [`DataFrame`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html#pandas.DataFrame)):

```ipython3
arr = xr.DataArray(
    np.random.randn(2, 3), coords=[("x", [10, 20]), ("y", ["a", "b", "c"])]
)
df = arr.to_pandas()
df
```

To perform the inverse operation of converting any pandas objects into a data
array with the same shape, simply use the [`DataArray`](../generated/xarray.DataArray.html.md#xarray.DataArray)
constructor:

```ipython3
xr.DataArray(df)
```

Both the `DataArray` and `Dataset` constructors directly convert pandas
objects into xarray objects with the same shape. This means that they
preserve all use of multi-indexes:

```ipython3
index = pd.MultiIndex.from_arrays(
    [["a", "a", "b"], [0, 1, 2]], names=["one", "two"]
)
df = pd.DataFrame({"x": 1, "y": 2}, index=index)
ds = xr.Dataset(df)
ds
```

However, you will need to set dimension names explicitly, either with the
`dims` argument on in the `DataArray` constructor or by calling
[`rename`](../generated/xarray.Dataset.rename.html.md#xarray.Dataset.rename) on the new object.

<a id="panel-transition"></a>

## Transitioning from pandas.Panel to xarray

`Panel`, pandas’ data structure for 3D arrays, was always a second class
data structure compared to the Series and DataFrame. To allow pandas
developers to focus more on its core functionality built around the
DataFrame, pandas removed `Panel` in favor of directing users who use
multi-dimensional arrays to xarray.

Xarray has most of `Panel`’s features, a more explicit API (particularly around
indexing), and the ability to scale to >3 dimensions with the same interface.

As discussed in the [data structures section of the docs](data-structures.html.md#data-structures), there are two primary data structures in
xarray: `DataArray` and `Dataset`. You can imagine a `DataArray` as a
n-dimensional pandas `Series` (i.e. a single typed array), and a `Dataset`
as the `DataFrame` equivalent (i.e. a dict of aligned `DataArray` objects).

So you can represent a Panel, in two ways:

- As a 3-dimensional `DataArray`,
- Or as a `Dataset` containing a number of 2-dimensional DataArray objects.

Let’s take a look:

```ipython3
data = np.random.default_rng(0).random((2, 3, 4))
items = list("ab")
major_axis = list("mno")
minor_axis = pd.date_range(start="2000", periods=4, name="date")
```

With old versions of pandas (prior to 0.25), this could stored in a `Panel`:

```python
pd.Panel(data, items, major_axis, minor_axis)
```

```none
<class 'pandas.core.panel.Panel'>
Dimensions: 2 (items) x 3 (major_axis) x 4 (minor_axis)
Items axis: a to b
Major_axis axis: m to o
Minor_axis axis: 2000-01-01 00:00:00 to 2000-01-04 00:00:00
```

To put this data in a `DataArray`, write:

```ipython3
array = xr.DataArray(data, [items, major_axis, minor_axis])
array
```

As you can see, there are three dimensions (each is also a coordinate). Two of
the axes of were unnamed, so have been assigned `dim_0` and `dim_1`
respectively, while the third retains its name `date`.

You can also easily convert this data into `Dataset`:

```ipython3
array.to_dataset(dim="dim_0")
```

Here, there are two data variables, each representing a DataFrame on panel’s
`items` axis, and labeled as such. Each variable is a 2D array of the
respective values along the `items` dimension.

While the xarray docs are relatively complete, a few items stand out for Panel users:

- A DataArray’s data is stored as a numpy array, and so can only contain a single
  type. As a result, a Panel that contains [`DataFrame`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html#pandas.DataFrame) objects
  with multiple types will be converted to `dtype=object`. A `Dataset` of
  multiple `DataArray` objects each with its own dtype will allow original
  types to be preserved.
- [Indexing](indexing.html.md#indexing) is similar to pandas, but more explicit and
  leverages xarray’s naming of dimensions.
- Because of those features, making much higher dimensional data is very
  practical.
- Variables in `Dataset` objects can use a subset of its dimensions. For
  example, you can have one dataset with Person x Score x Time, and another with
  Person x Score.
- You can use coordinates are used for both dimensions and for variables which
  \_label_ the data variables, so you could have a coordinate Age, that labelled
  the Person dimension of a Dataset of Person x Score x Time.

While xarray may take some getting used to, it’s worth it! If anything is unclear,
please [post an issue on GitHub](https://github.com/pydata/xarray) or
[StackOverflow](https://stackoverflow.com/questions/tagged/python-xarray),
and we’ll endeavor to respond to the specific case or improve the general docs.