Microarray::DataMatrix::BigDataMatrix - abstraction to matrix that won't fit in memory
bigDataMatrix is an abstract class, which provides as abstraction to a matrix of data that is too large to fit into memory. It should not be subclassed by concrete subclasses. Instead, the subclass anySizeDataMatrix, can be subclassed with concrete subclasses, which will provide abstractions to dataMatrices stored in particular file formats, such as pcl files.
Internally, bigDataMatrix simply keeps track of which rows and columns are still valid. As it runs filters, or transformations on data, it creates temp files, which contain the result of such an operation. It keeps track of which rows and columns in the latest temp file are valid, and also to which rows and columns in the original file, used for object construction, that they map. Then, when it comes time to dump data to disk, it is able to instruct a concrete subclass what the original index of a row or column was, so that the concrete subclass can print out the appropriate meta data.
As bigDataMatrix is an abstract class, it has no constructor. However, the subclass, anySizeDataMatrix, once it has determined that a matrix will not fit into memory, MUST call the _init method, which is described below.
This protected method will read determine how many rows and columns there are in the initial matrix. It MUST be called during initialization of a subclass object, before any other methods can be called by the client (in practice it is called from anyDataMatrix).
Usage:
$self->_init;
or:
$self->SUPER::_init;
This private method receives the number of rows that the matrix has, and simply records each one of them as valid. In addition it sets up a map of the current row positions in a temp file, to their original positions in the file used for construction.
$self->__validateRows($numRows);
This method receives the number of columns that the matrix has, and simply records each one of them as valid. In addition it sets up a map of the column positions in the original file to their current positions, and a reverse map of the columns current positions, to their positions in the original matrix file used for object construction.
Usage :
$self->__validateColumns($numColumns);
This private mutator method makes a row invalid. We actually do the invalidation in the super class, but have to use this method to call the superclass method, so that we mark the row index from the original file as invalid, rather than marking as invalid the row index based on the rows current index. We do this by translating the current row number to its original row index.
$self->__invalidateMatrixRow($num);
This private method returns the original row number to which a row in the current file now corresponds.
my $origRow = $self->__origRow($row);
This private mutator method makes a column invalid. We actually do the invalidation in the super class, but have to use this method to call the superclass method, so that we mark the columns index from the original file as invalid, rather than marking as invalid the column index based on the current index of a column. We do this by translating the current column number to its original.
$self->__invalidateMatrixColumn($column);
This method returns the original column number to which a column in the current file now corresponds.
my $column = $self->__origColumn($column);
This private method remaps a line in a tmp file to the line it corresponds to in the original file. It receives the current tmp file line, and the number of lines that have been removed before it. It then remaps the current row to point to the original row that the current row, $numFilteredLines ahead of it, is pointing to.
$self->__remapRow($numRowsPrinted, $numFilteredLines);
This private method remaps a column in a tmp file to the column it corresponds to in the original file. It receives the current file column index. It then remaps the current column to point to the original column that the current column, $numFilteredColumns ahead of it, is pointing to.
$self->__remapColumn($numColumnsPrinted, $numFilteredColumns);
This method remaps all of the columns in the current tmp file with respect to their column index in the original file. This method must be called after carrying out any operation that may reduce the number of columns in the matrix.
$self->__remapColumns;
This private accessor returns a boolean to indicate whether a given row in the tmp file is still valid (ie has not been filtered out). It does this by mapping the row to what its original index was in the file used to instantiate the matrix, and then determining whether that index is still valid.
if ($self->__currentRowIsValid($row)){ # blah }
This private accessor returns a boolean to indicate whether a given column in the tmpfile is still valid (ie has not been filtered out). It does this by mapping the column to what its original index was in the file used to instantiate the matrix, and then determining whether that index is still valid.
if ($self->__currentColumnIsValid($column)){ # blah }
This private method returns an reference to an array that contains the indices of the columns that are currently valid in the present file.
my $validColumnsArrayRef = $self->__currentValidColumnsArrayRef;
This private method returns an reference to an array that contains the indices of the rows that are currently valid in the present file
foreach (@{$self->__currentValidRowsArrayRef}){ # blah }
This method returns the index where a column in the original file, now maps in the current temp file.
my $col = $self->__currentColumnForOrig($column);
This private method returns a handle to a tmpfile. If given a 'new' argument, which will be passed to the tmpFile method, the file will have its associated number increased by one - ie a new file will be used. Otherwise, a previously created file will be opened.
my $fh = $self->__tmpFileHandle(new=>1);
This private method returns the name of a tmpfile. If given the 'new' argument then the name will be of a new file. Otherwise it will be of the last generated tmpfile.
my $tmpFile = $self->__tmpFile(new=>1);
This method returns the number of the temp file. Generally this number should be incremented by 1 each time a new tmp file is created.
my $num = $self->__tmpNum;
This method allows the tmp num to be set.
$self->__setTmpNum($num);
This private method is used for centering the columns of a dataset by the mean value.
my $largestVal = $self->__centerColumns_mean($lineEnding, $numColumnsToReport);
This method is used for centering the columns of a dataset by the median value.
Because we need to have a sorted list of values to calculate the median, we are going to have to read through all the data multiple times (potentially), to avoid having to have too much data in memory.
my $largestVal = $self->__centerColumns_median($lineEnding, $numColumnsToReport);
This method retrieves data from the dataMatrix through the subclass, subtracts the column average from each value, then, through the subclass methods writes out a new file.
$self->__subtractColumnAverages(\@medians, $lineEnding, $numColumnsToReport);
This private method filters out rows that do not have a count for some particular property above or equal to a threshold. It accepts a hash reference, that hashes the row number to a count, and a threshold value. Note that not all rows are necessarily entered into the hash, so this method iterates over all rows, and checks each valid one for its count in the hash, then invalidates those with too low a count.
$self->__filterRowsByCount(\%count, $numColumns);
This method calculates the standard deviations for each valid column, and returns references to two hashes. Both have the column index as the key, and one has the standard deviation as the values, the other has the column means as the values.
mean = Sum of values/n std dev = square root (((n * sum of (x^2)) - (sum of x)^2)/n(n-1))
my ($stddevHashRef, $meansHashRef) = $self->__validColumnsStdDevAndMeanHashRefs($lineEnding);
Note: These methods provide the backend nuts and bolts for a transformation or filtering. They should only be called by the immediate subclass, anySizeDataMatrix, and not directly by the concrete subclasses of anySizeDataMatrix. In addition, note that the companion smallDataMatrix must (and does) provide identical interfaces to these methods (obviously with different underlying implementations), such that anySizeDataMatrix can call the methods without regard to the size of the underlying matrix.
This protected method centers each column of data, and returns the largest absolute value that was used in the centering. The caller of the method must specify whether to center by means or medians.
$self->_centerColumns('mean', $lineEnding, $numColumnsToReport);
This protected method actually centers the row data, by calculating the average (mean or nedian, depending on what was requested) for each row, and then subtracting that value from each valid datapoint in the row.
$self->_centerRows('median', $lineEnding, $numRowsToReport);
This protected method invalidates rows that do not have greater than the requested percentage of present data.
$self->_filterRowsByPercentPresentData($percent, $lineEnding, $numRowsToReport);
This protected method invalidates columns that do not have greater than the requested percentage of present data.
$self->_filterColumnsByPercentPresentData($percent, $lineEnding, $numColumnsToReport);
NB: THIS METHOD HAS NOT YET BEEN IMPLEMENTED FOR BIG DATAMATRICES
This protected method filters out rows based on their column percentile, when all data are known to be in memory, and optionally allows for the percentiles of each datapoint to be displayed in the output file.
$self->_filterRowsOnColumnPercentile($lineEnding, $numColumnsToReport, $percentile, $numColumns, $showPercentile);
This protected method will filter out rows whose values do not deviate from the column mean by a specified number of standard deviations, in at least numColumns columns.
$self->_filterRowsOnColumnDeviation($lineEnding, $numRowsToReport, $deviations, $numColumns);
This protected method filters out rows whose values do not pass a specified criterion, in at least numColumns columns.
$self->_filterRowsOnValues($value, $method, $lineEnding, $numRowsToReport, $numColumns);
This protected method filters out rows based on whether the vector that their values define has a length of greater than the specified length.
$self->_filterRowsOnVectorLength($requiredLength, $lineEnding, $numRowsToReport);
This method log transforms the contents of the data matrix, using the specified base for the log transformation.
$self->_logTransformData($logBase, $lineEnding, $numRowsToReport);
This protected method scales the data for particular columns as specified by the client, when all data are in memory.
$self->_scaleColumnData($columnsToFactorsHashRef, $lineEnding, $numColumnsToReport);
This method dumps the current contents of the dataMatrix object to a file, either whose name was provided as a single argument, or to a file whose name was used to construct the object.
$self->dumpData($file);
Gavin Sherlock
sherlock@genome.stanford.edu
To install Microarray::Config, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Microarray::Config
CPAN shell
perl -MCPAN -e shell install Microarray::Config
For more information on module installation, please visit the detailed CPAN module installation guide.