`transpose` doesn't seem to exist?

[gfarrell]

In pandas you can transpose the dataframe in case your source data is the wrong orientation. This is mentioned in various places in the dataframe documentation, e.g. https://dataframe.readthedocs.io/en/latest/dataframes_in_haskell.html#api-design mentions a transpose :: DataFrame -> DataFrame function and it's mentioned in passing in #144 but I cannot, for the life of me, find where this function is supposedly defined. It appears not to be in the library (at least from my searching through the documentation on Hackage).

Am I going mad?

[mchav]

Not yet! We've been dreading working on these schema destroying operations like transpose and pivot. There are obvious ways to do them but they are slow and memory hungry. Do you need this prioritized?

[gfarrell]

Don’t prioritise for my sake — was just playing with some very annoying data. Am curious about what makes it hard (as have no intuition for that in the context of DF).

[daikonradish]

It's schema destroying in the sense that dataframes represent a mapping from a column name to a column type. For example, you could have a column of Ints representing the age of a person, and a column of Text representing their names.

This allows us to have several advantages when we do computations.

For float, int, bool columns -- they are optimally represented with an Unboxed Vector, which means that we can transform such columns at close to the speed of C.

Once the columns are transposed, we lose the ability to assert such invariants about the dataset. Each column must be represented as a vector of heterogenous type, which is not only quite hard to write in Haskell, but also quite slow, as things fragment on the heap and stack.

For transposition, if you need matrix transposition within torch, Hasktorch exports transpose, which is fast as it is simply a binding to torch's transpose.

[gfarrell]

I see -- my use-case is simply that the data have come in the wrong shape, so rows should be columns and vice-versa. Hence I'd be happy with an "import transposed" option which transposes the input data, then does all this type mapping (which I think is a much simpler problem based on what you're saying).

[gfarrell]

I considered just writing a utility to transform the input this way and then pass it into DF.

[daikonradish]

ah, yes, in that case, transpose won't help either (and that's why we've deferred on writing it), because we haven't decided what the sensible behavior should be in Haskell. When you read in a dataframe, the library makes assumptions on types, so it reads the entire csv in and then infers, say, that age should be Int and name should be Text.

However, when you transpose that information, each column then has type Any. When you transpose that again, age is Any.

So it would be much easier to re-export the data in conformant csv format!