APPENDIX E: STSDAS Tables STSDAS tables are binary files that contain data in row and column format. Each column has a name, data type, print format, and unit. All the values in a given column are of the same data type, but different columns may have different data types. The column name should be unique within a table. The print format may be used to display the values but does not affect the way the values are stored in the table. The units string may contain any information that will fit; calling it "units" is just a suggestion. A table may also contain header parameters in a format similar to FITS header keywords. The data types supported for tables are double precision real, single precision real, integer, boolean, and text strings. Values are stored in the table file in the host machine's binary format. Elements that have not been assigned values or that have been set to "undefined" are flagged as such in the table. The object library specified to xc as -ltbtables contains all the spp-callable table I/O routines. The include file tbset.h defines parameters for getting such information as the number of rows or columns in a table. Some items may also be set. The maximum lengths of column names and similar values are also specified in that file. Further details are given below. The tbpset routine is used to set parameter values, and the integer function tbpsta returns values. A table with more than one column is a 2-D array of values. A 2-D array can be stored in the file in row or column ordered format. That is, as you step from word to word in the file, you could be stepping along a row or down a column. Both options are supported for STSDAS tables. Simple text files in row and column format can also be accessed as tables by the STSDAS table I/O routines. The file name for a binary table must include an extension, with tab as the default. A text table, on the other hand, need not have an extension. STDIN and STDOUT may be used for input and output text tables. The table interface includes routines for accessing table files, columns, header parameters, table parameters, and table data. The name of each routine begins with "tb", the next letter indicates what type of object is involved (row, column, parameter, etc.), and the last three letters specify what is to be done (e.g., open, close, get, put). For example, tbtopn opens a table. The third letter ("t") implies that the routine applies to a table as a whole, and "opn" means "open". Similarly, tbtclo closes a table. For some routines the last letter indicates the data type of the input or output buffer. For example, tbegtr operates on a table element ("e") to get ("gt") an element, and the output buffer is of type real ("r"). The corresponding "put" routine is tbeptr. Table E.1 is a list of third letters and what they refer to: Letter Object Examples of use -------------------------------------------------------------------------- t Table file Open, close, get table name p Table parameter Number of rows, number of columns h Header parameter Get or put header parameter c Column Find, create, get or put column r Row Get or put values in a row e Element Get or put a single value -------------------------------------------------------------------------- Table E.1: Table I/O Procedure Naming Conventions. Procedure Description -------------------------------------------------------------------------- tp = tbtopn (tablename, iomode, template) Initialize (and open the table if not NEW_FILE or NEW_COPY) tbtcre (tp) Create new table (after initializing with tbtopn) tbtclo (tp) Close a table -------------------------------------------------------------------------- Table E.2: Procedures to Open and Close Tables. Example E.1 reads all values from one table column and prints the values that are defined. If this were in a file called test.x, it could be compiled and linked by typing: xc -p stsdas test.x -ltbtables. ----------------------------------------------------------------------- task test include # defines TBL_NROWS, SZ_COLNAME, etc procedure test() pointer tp # pointer to table descriptor pointer cp # pointer to column descriptor char intable[SZ_FNAME] # table name char colname[SZ_COLNAME] # column name real value # a single value from a table element int nrows # number of rows in table int row # loop index for row number pointer tbtopn() int tbpsta() begin call clgstr ("intable", intable, SZ_FNAME) call clgstr ("colname", colname, SZ_FNAME) tp = tbtopn (intable, READ_ONLY, NULL) # open the table call tbcfnd (tp, colname, cp, 1) # find the column in the table if (cp == NULL) { call tbtclo (tp) call error (1, "column not found") } nrows = tbpsta (tp, TBL_NROWS) do row = 1, nrows { call tbegtr (tp, cp, row, value) # get value in current row if (!IS_INDEF(value)) { # is the value defined? call printf ("%14.6g\n") call pargr (value) } } call tbtclo (tp) # close the table end ----------------------------------------------------------------------- Example E.1: Table I/O Example. Procedure Description -------------------------------------------------------------------------- tbcdef (tp, colptr, colname, colunits, colfmt, datatype, lendata, numcols) Define columns tbcfnd (tp, colname, colptr, numcols) Find a column from its name tbcinf (colptr, colnum, colname, colunits, colfmt, datatype, lendata, lenfmt) Get information about a column int = tbcigi (colptr, param) Get specific info about a numeric column (e.g. name or data type) tbcigt (colptr, param, outstr, maxch) Get specific info about a string column (e.g. name or data type) -------------------------------------------------------------------------- Table E.3: Procedures Dealing with Columns. Procedure Description -------------------------------------------------------------------------- tbtcpy (inname, outname) Copy a table tbtdel (tablename) Delete a table tbtren (oldname, newname) Rename a table int = tbtacc (tablename) Test for the existence of a table tbtext (inname, outname, maxch) Append default extension (if it's not already there) tbtnam (tp, tblname, maxch) Get the name (including extension) of the table tbtflu (tp) Flush FIO buffer for table -------------------------------------------------------------------------- Table E.4: Table File Operations. Reading and Writing Data Three sets of get and put routines are provided for accessing table data. The "tbe..." routines get or put single elements; that is, values at a specified row and column. The "tbr..." routines get or put one or more elements in a single row. The "tbc..." routines get or put values in a single column over a range of rows. The last (sixth) letter of each routine name specifies the buffer data type: "t" for a text string, "b" for boolean, "i" for integer, "r" for real, and "d" for double precision. The data type of the buffer does not need to be the same as the data type of the table column; the table I/O routines convert data type when the column and buffer do not match. The tbrgtT and tbcgtT routines return a boolean array that indicates whether the table elements gotten are undefined. A true value means the table element is undefined. The tbegtT routine returns the data type-specific INDEF value when the table element is undefined. When writing values into a table, values may be set to undefined by calling tbrudf. If a row exists, but no value has ever been written to a particular column in that row, the element at that row and column will automatically be undefined; that is, it is not necessary to call tbrudf. A row exists if a value has been put into any column in that row or into a subsequent row (larger row number). Procedure Data Types Description -------------------------------------------------------------------------- tbegtT (tp, colptr, rownum, buffer) b i r d Get a numeric value from the table tbegtt (tp, colptr, rownum, buffer, maxch) Get a string value from the table tbeptT (tp, colptr, rownum, buffer) t b i r d Put a value into the table tbrgtT (tp, colptr, buffer, nullflag, numcols, rownum) b i r d Get numeric values from a row tbrgtt (tp, colptr, buffer, nullflag, lenstr, numcols, rownum) Get string values from a row tbrptT (tp, colptr, buffer, numcols, rownum) b i r d Put numeric values into a row tbrptt (tp, colptr, buffer, lenstr, numcols, rownum) Put string values into a row tbcgtT (tp, colptr, buffer, nullflag, firstrow, lastrow) b i r d Get numeric values from a column tbcgtt (tp, colptr, buffer, nullflag, lenstr, firstrow, lastrow) Get string values from a column tbcptT (tp, colptr, buffer, firstrow, lastrow) b i r d Put numeric values into a column tbcptt (tp, colptr, buffer, lenstr, firstrow, lastrow) Put string values into a column tbrudf (tp, colptr, numcols, rownum) Set values in a row to undefined -------------------------------------------------------------------------- Table E.5: Table Get and Put Procedures. Example E.2 gets two values from each row of a table and copies them to another table if neither value is undefined. A double-precision buffer is used so that data of any numerical type will be copied without loss of precision. ------------------------------------------------------------------------- include define NCOLS 2 # number of columns to get procedure test() pointer sp # stack pointer pointer intable, outtable # scratch for table names pointer ira, idec # scratch for arrays of input values pointer ora, odec # scratch for arrays of output values pointer ra_flag # scratch for array of null flags pointer dec_flag # scratch for array of null flags char cname[SZ_COLNAME,NCOLS] # column names pointer itp, otp # pointers to table descriptors pointer icp[NCOLS] # pointers to column descriptors in input pointer ocp[NCOLS] # pointers to column descriptors in output int inrows, onrows # number of rows in input, output tables int irow # loop index for row number in input table int orow # row number in output table int i # loop index bool nullflag[NCOLS] # null flags for getting info from a row bool bad # true if any element of nullflag is true double value[NCOLS] # values gotten from a table pointer tbtopn() int tbpsta() begin # Allocate scratch space for table names. We'll allocate space # for column values later, after we know the size of the table. call smark (sp) call salloc (intable, SZ_FNAME, TY_CHAR) call salloc (outtable, SZ_FNAME, TY_CHAR) # Get table names. call clgstr ("intable", Memc[intable], SZ_FNAME) call clgstr ("outtable", Memc[outtable], SZ_FNAME) # Get column names. call clgstr ("ra_col", cname[1,1], SZ_COLNAME) call clgstr ("dec_col", cname[1,2], SZ_COLNAME) # Open input table. itp = tbtopn (Memc[intable], READ_ONLY, NULL) # Find columns in input table. Check if they were found. call tbcfnd (itp, cname, icp, NCOLS) if (icp[1] == NULL || icp[2] == NULL) { call tbtclo (itp) call error (1, "column not found") } ------------------------------------------------------------------------------ Example E.2: Copying Columns. ------------------------------------------------------------------------------ # Create an output table with the same columns as the input table. otp = tbtopn (Memc[outtable], NEW_COPY, itp) call tbtcre (otp) # Copy header parameters from input to output. call tbhcal (itp, otp) # Find columns in output table. They will be there since they were # in the input table. call tbcfnd (otp, cname, ocp, NCOLS) # There will be fewer rows in the output table if the columns # we're interested in contain undefined elements. inrows = tbpsta (itp, TBL_NROWS) # Here are three different ways of copying the values. # 1. Copy element by element. orow = 0 do irow = 1, inrows { call tbegtd (itp, icp[1], irow, value[1]) call tbegtd (itp, icp[2], irow, value[2]) if (!IS_INDEFD(value[1]) && !IS_INDEFD(value[2])) { orow = orow + 1 call tbeptd (otp, ocp[1], orow, value[1]) call tbeptd (otp, ocp[2], orow, value[2]) } } # 2. Use the get-row and put-row routines. This will copy # any number of columns, one row at a time. orow = 0 do irow = 1, inrows { call tbrgtd (itp, icp, value, nullflag, NCOLS, irow) bad = false do i = 1, NCOLS if (nullflag[i]) bad = true if (!bad) { orow = orow + 1 call tbrptd (otp, ocp, value, NCOLS, orow) } } ------------------------------------------------------------------------------ Example 5.2 (Continued): Copying Columns. ------------------------------------------------------------------------------ # 3. Use the get-column and put-column routines. call salloc (ira, inrows, TY_DOUBLE) call salloc (idec, inrows, TY_DOUBLE) call salloc (ra_flag, inrows, TY_BOOL) call salloc (dec_flag, inrows, TY_BOOL) call salloc (ora, inrows, TY_DOUBLE) # possibly more than we need call salloc (odec, inrows, TY_DOUBLE) call tbcgtd (itp, icp[1], Memd[ira], Memb[ra_flag], 1, inrows) call tbcgtd (itp, icp[2], Memd[idec], Memb[dec_flag], 1, inrows) # Note that irow and orow are zero indexed in this loop. orow = -1 do irow = 0, inrows-1 { if (!Memb[ra_flag+irow] && !Memb[dec_flag+irow]) { orow = orow + 1 Memd[ora+orow] = Memd[ira+irow] Memd[odec+orow] = Memd[idec+irow] } } onrows = orow + 1 # number of rows in output table if (orow > 0) { call tbcptd (otp, ocp[1], Memd[ora], 1, onrows) call tbcptd (otp, ocp[2], Memd[odec], 1, onrows) } # Done. Three times, even. call tbtclo (itp) call tbtclo (otp) call sfree (sp) end ------------------------------------------------------------------------------ Example 5.2 (Continued): Copying Columns. Header Parameters Tables may contain header parameters consisting of a keyword name, data type flag, and a value. These are stored in the table as text strings. These parameters are not used for information such as the number of rows or columns, and the table I/O routines do not use header parameters when getting or putting table elements. The same data types are supported for header parameters as for table data, and type conversion is performed, except that a value stored as a text string may only be gotten as text, not as numeric or boolean. The distinction between adding and putting values is the same as for image header keywords. You can call tbhptT to put a header parameter only if that parameter already exists in the table, but you can call tbhadT to either add a new header parameter or replace an existing one. In contrast to the imio interface, when you open a table NEW_COPY, the header parameters are not copied. Procedure Data Types Description -------------------------------------------------------------------------- value = tbhgtT (tp, param) b d i r Get a numeric header parameter tbhgtt (tp, param, text, maxch) Get a string header parameter tbhadT (tp, param, value) t b d i r Add a new header parameter or replace existing one tbhptT (tp, param, value) t b i r d Replace an existing header parameter tbhcal (itp, otp) Copy all header parameters tbhgnp (tp, parnum, keyword, dtype, str) Get Nth header parameter as a string -------------------------------------------------------------------------- Table E.6: Header Parameter Procedures. The tbset.h Include File This section describes the include file tbset.h. In most situations the only parameters that will be needed are SZ_COLNAME and TBL_NROWS. These three are used for declaring the sizes of char variables for column names, units, and print formats. o SZ_COLNAME - Maximum length of a column name o SZ_COLUNITS - Maximum length of string for units o SZ_COLFMT - Maximum length for print format The next four parameters may be read by tbpsta but may not be set: Parameter Meaning -------------------------------------------------------------------------- TBL_NROWS Number of rows written to TBL_NCOLS Number of columns defined TBL_ROWLEN_USED Amount of row length used (unit = size of single precision) TBL_NPAR Number of user parameters -------------------------------------------------------------------------- Table E.7: Non-settable Parameters Read by tbpsta. These may be set by tbpset or read by tbpsta. Parameters TBL_ROWLEN and TBL_INCR_ROWLEN are relevant only to row-ordered tables, while TBL_ALLROWS and TBL_INCR_ALLROWS are relevant only to column-ordered tables. TBL_ROWLEN is for setting the row length to a specific value. In contrast, TBL_INCR_ROWLEN is used to increase the row length by the specified amount over its current value, whatever that may be. The latter is more useful. When creating a new table, we suggest the following procedure for a row-ordered table. After calling tbtopn, define columns using tbcdef. Then the row length will be sufficient for the columns that have been defined. If you will need to define more columns after the table has been created, you can call tbpset with TBL_INCL_ROWLEN to preallocate the needed space before creating the table with tbtcre. The numerical value would be one for each single-precision or integer column, and two for each double-precision column. For character strings, divide the maximum string length by the number of bytes in a single-precision variable and round up. Parameter Meaning -------------------------------------------------------------------------- TBL_ROWLEN Row length to allocate (units are the size of a single-precision) TBL_INCR_ROWLEN Increase row length (in single-precision units) TBL_ALLROWS Number of rows to allocate TBL_INCR_ALLROWS Increase number of allocated rows TBL_WHTYPE Type of table? (see below) TBL_MAXPAR Maximum number of user parameters TBL_MAXCOLS Maximum number of columns -------------------------------------------------------------------------- Table E.8: Table Parameters That Can be Read or Set. The table type as set or read using TBL_WHTYPE is defined byt the parameters in Table E.9. Parameter Meaning -------------------------------------------------------------------------- TBL_TYPE_S_ROW Row-ordered binary table TBL_TYPE_S_COL Column-ordered binary table TBL_TYPE_TEXT Text file -------------------------------------------------------------------------- Table E.9: Table Types. The parameters described in Table E.10 have to do with the file size and file I/O buffer size. Parameter Meaning -------------------------------------------------------------------------- TBL_ADVICE Set RANDOM or SEQUENTIAL TBL_BUFSIZE Get buffer size in characters TBL_DATA_SIZE Get size of table data in characters ------------------------------------------------------------------------- Table E.10: Table Size and File I/O Buffer Size. The parameters are for getting information about a column using tbcigt or tbcigi. Parameter Meaning ---------------------------------------------------------------------- TBL_COL_NAME Column name TBL_COL_UNITS Units for column TBL_COL_FMT Print format for displaying values TBL_COL_DATATYPE Data type (-n for character string) TBL_COL_NUMBER Column number TBL_COL_FMTLEN Length for printing using print format TBL_COL_LENDATA Number of elements if colum is an array -------------------------------------------------------------------------- Table E.11: Getting Column Information. Procedure Description -------------------------------------------------------------------------- tbpset (tp, setwhat, value) Set a table parameter int = tbpsta (tp, param) Get the value of a table parameter (e.g. number of rows) int = tbcigi (colptr, param ) Get information about column (integer) tbcigt (colptr, param, outstr maxch) Get information about column (string) -------------------------------------------------------------------------- Table E.12: Table Parameter Procedures. Print Formats The print format is used by such tasks as tprint, tedit, and tread to determine how the column values are to be displayed. The earlier statement that the print format does not affect the way the values are stored in the table is really only true for binary tables. For output (or read-write) text tables the print format is actually used to write the file, so it is critical with regard to the precision of the data values. Most of the ordinary Fortran formats are supported for tables. SPP formats are discussed in the fmtio section of this document. The only SPP print formats that are not allowed are those that are simply irrelevant, such as t, w, and z. The field width may not be zero, however. The procedure tbbftp may be used to convert a user-supplied Fortran style format to an SPP style format. Table E.13 is a list of the default print format for each data type, given in both SPP style and Fortran style. Data type SPP Fortran -------------------------------------------------------------------------- real %15.7g G15.7 double prec %25.16g G25.16 integer %11d I11 boolean %6b L6 text string %-ns A-n -------------------------------------------------------------------------- Table E.13: Default Print Formats. For character strings "n" is the string size as given when the column was defined. The minus sign means that the string will be left justified. While a format such as "A-12" is not available in standard Fortran, the tbbftp routine will convert it to "%-12s". SPP formats and Fortran equivalents that are supported for tables are listed in this table. The syntax is %w.dC (SPP style) or Cw.d (Fortran style), where w is the field width, d is the number of decimal places (or precision for g format), and C is the format code as given in the left column below. When giving a format in Fortran style, use the format code given in the second column; these are shown in upper case but may also be given in lower case. Note that H and M are not standard Fortran formats; in particular, H is not interpreted as Hollerith. SPP Fortran Meaning -------------------------------------------------------------------------- b L Boolean "yes" or "no" d I Integer, displayed in decimal x Z Integer, displayed in hexadecimal e E or D Exponential format f F Floating point g G Use F or E as appropriate h H HH:MM:SS.d (sexagesimal) m M HH:MM.d (sexagesimal) s A Character string -------------------------------------------------------------------------- Table E.14: Table Print Formats. Table Utilities Table E.15 lists some table utility procedures. These permit operating on entire columns or rows and performing other funtions on the table as a whole. Note also that the tbtables package of tasks in the STSDAS external package that allows flexible and sophisticated manipulation of existing tables without writing any code. These include such database-related functions as extracting selected rows based on the value of particular fields, extracting given columns by name, printing a report from a table or editing a table in-place. See help tbtables for a list of the tasks and a brief description of each. Procedure Description -------------------------------------------------------------------------- tbtchs (tp, maxpar, maxcols, rowlen, allrows) Change allocated space of any/all portions of a table tbrcpy (itp, otp, irownum, orownum) Copy an entire row (only for tables with identical columns) tbrcsc (itp, otp, icptr, ocptr, irownum, orownum, ncols) Copy a row, but copy only selected columns tbrswp (tp, row1, row2) Swap two rows tbtsrt (tp, numcols, colptr, fold, nindex, index) Sort an index for the table rows tbrdel (tp, firstrow, lastrow) Delete a range of rows tbrnll (tp, firstrow, lastrow) Set all columns in a range of rows to INDEF tbcnam (tp, colptr, colname) Change the name of a column tbcfmt (tp, colptr, colfmt) Change the format for printing a column tbcnit (tp, colptr, colunits) Change the units for a column colptr = tbcnum (tp, colnum) Get the column pointer from the column number -------------------------------------------------------------------------- Table E.15: Table Utility Procedures.