Pandas - retrieving HDF5 columns and memory usage

HDFStore in table format is a row oriented store. When selecting the query indexes on the rows, but for each row you get every column. selecting a subset of columns does a reindex at the end.

There are several ways to approach this:

• use a column store, like bcolz; this is currently not implemented by PyTables so this would involve quite a bit of work
• chunk thru the table, see here and concat at the end - this will use constant memory
• store as a fixed format - this is a more efficient storage format so will use less memory (but cannot be appended)
• create your own column store-like by storing to multiple sub tables and use select_as_multiple see here

which options you choose depend on the nature of your data access

note: you may not want to have all of the columns as data_columns unless you are really going to select from the all (you can only query ON a data_column or an index) this will make store/query faster