Beginner's IDL - Data Input and Output




The objective of this chapter is to provide a more in-depth look at a specific topic in IDL. Here we will look at how to read and write data from IDL.



Brief Recap from Beginner's IDL I


Before diving into data input and output in IDL, consider the methods we discussed before.


How to read and write files

Read data


How to read in a table with 2 columns, 5 rows

Method 1:
IDL> data=fltarr(2,5) ; create the data array
IDL> openr,lun,'in_array.dat',/get_lun ; open the data file
IDL> readf,lun,data ; read data from the file
IDL> close,/all ; close the data file
IDL> help,data ; check the data array
DATA FLOAT = Array[2, 5]
IDL> print,data
1.00000 5.00000
2.00000 6.00000
3.00000 7.00000
4.00000 8.00000
5.00000 9.00000
IDL> x=reform(data(0,*)) ; separate the columns
IDL> y=reform(data(1,*))
IDL> print,x
1.00000 2.00000 3.00000 4.00000 5.00000
IDL> print,y
5.00000 6.00000 7.00000 8.00000 9.00000

Method 2:
IDL> x=fltarr(5) ; create the x data array
IDL> y=fltarr(5) ; create the y data array
IDL> openr,lun,'in_array.dat',/get_lun ; open the data file
IDL> .run ; execute a group of control statement
- for j=0,4 do begin ;start of the loop
- readf,lun,tmp1,tmp2 ; read the j-th row
- x(j)=tmp1 ; put the data into the x array
- y(j)=tmp2 ; put the data into the y array
- endfor ; end of the for loop
- end ; end of the control group
% Compiled module: $MAIN$.
IDL> print,x
1.00000 2.00000 3.00000 4.00000 5.00000
IDL> print,y
5.00000 6.00000 7.00000 8.00000 9.00000

How to read in unformatted binary data

IDL> openr, lun, 'galaxy.dat', /get_lun ; open the data file
IDL> image=bytarr(256,256) ; create a 256x256 byte array
IDL> readu, lun, image ; read in the data
IDL> free_lun, lun
; close the file

How to read in HST data

Check out the widget_olh

Write data


Replace openr (open-read) with openw (open-write), and replace readf (read-file) with printf (print-file).

How to write the 2 by 5 array back:

IDL> openw,lun,'out_array.dat',/get_lun
IDL> printf, lun, data
IDL> Free_lun, lun


or

IDL> openw,lun,'out_array.dat',/get_lun
IDL> printf, lun, x, y
IDL> Free_lun, lun

We will take a closer look at reading and writing data in IDL here.


IDL can read many different file formats


As the heading suggests, IDL can read and write in many different file formats. There are IDL library routines that are readily available for use in reading some common formats, such as PICT, JPEG, TIFF, etc. Also, there are other routines written by others that would allow you to read FITS file as well as IRAF images. Table 1 here summarizes the various file formats and the corresponding IDL routines to use for reading the image data.
File Format Routine to Read Routine to Write
BMP Read_BMP Write_BMP
GIF Read_GIF Write_GIF
Interfile (v 3.3) Read_Interfile None
JPEG Read_JPEG Write_JPEG
PICT Read_PICT Write_PICT
PGM/PPM Read_PPM Write_PPM
PostScript None Set PS Graphics Device
Sun Rasterfiles Read_SRF Write_SRF
SYLK Read_SYLK Write_SYLK
TIFF TIFF_Read TIFF_Write
Wavefront Advanced Visualizer (WAVE) Read_WAVE Write_WAVE
X11-Bitmap Read_X11_Bitmap None
XWD Read_XWD None
CDF see CDF library see CDF library
netCDF see netCDF library see netCDF library
HDF See HDF library see HDF library

Table 1. File formats and routines to read and write them.


To change directory

IDL> cd,'/data/wombat1/IDL_course'



Reading and writing files in general


There are three steps in reading and writing files:


Step 1: Open a file for read, write, or update

There are three commands for opening a file:
OpenR Open a file for reading
OpenW Open a file for writing
OpenU Open a file for updating


The form of these commands:

IDL> OpenR, lun, 'filename'
or
IDL> OpenW, lun, 'filename'
or
IDL> OpenU, lun, 'filename'


where lun is a logical unit number and filename is a case sensitive name of the file you would like to open. You can also use a string variable instead.

IDL> NameOfFile = 'datafile.dat'
IDL> OpenR, lun, NameOfFile


About the Logical Unit Number (lun)

The Logical Unit Number (lun) is basically an index number assigned to an open file, and you can use this index number to refer to the file. In a way, it is like a pointer. The Open statement is to associate a filename with a lun. Three luns are reserved by the system and not available to the user:
0 The standard input stream
-1 The standard output stream
-2 The standard error stream

There are 128 other logical unit numbers that you can use:
1-99 You can use directly, but you will have to keep track of them yourself
100-128 You can use through the Get_LUN and Free_LUN procedures.


To use the lun between 1 and 99:

IDL> OpenR, 25, NameOfFile


Once a lun is assigned to a file by you, it cannot be reassigned until the file is closed by you. To close a file, simply use the CLOSE statememt:

IDL> Close, 25


To use the lun between 100-128, you will have to use the Get_LUN and Free_LUN procedures. The advantage of using the two procedures is that you will not need to keep track of the usage of the luns. As long as you do not use up all the luns, you will be guaranteed a valid lun return by the procedure. To open and close a file:

IDL> Get_LUN, lun
IDL> OpenR, lun, NameOfFile
IDL> ... do something...
IDL> Free_LUN, lun



STEP 2:

Reading Formatted Data


IDL can read files of with the following type of data formatting:

Free Format A free format file uses either commas or whitespace (tabs and spaces) to distinguish each element in the file
Explicit Format An explicit format file is formatted according to tules specified in a format statement. The IDL format statement is similar to the kind of format statement you would write in a FORTRAN program.


There are three commands for reading free formatted data:

Read Reads free format input from standard input, usually the keyboard.
ReadF Reads free format input from a file.
ReadS Reads free format input from a string variable.


Rules for reading free format data:

  1. Input data must be separated by commas or whitespace.
  2. Input is performed on scalar variables. Arrays and structures are treated as collections of scalar variables.
  3. If the current input line is empty, and there are still variables left requiring input, read another line.
  4. If the current input line is not empty, but there are no variables left requiring input, ignore the remainder of the line.
  5. Convert data into the data type expected by the variable.
  6. If reading into a string variable, all characters remaining on the current line are read into the variable.

So to read free format data:

IDL> array = IntArr(5)
IDL> Read, array



To read explicit format data:

IDL> ReadF, lun, var1, var2, Format='(3(8(F6.2,X)), 10I5)'


Refer to the IDL Reference for the format specifiers.


Examples of Reading Data


1. To read in a 3 column by 2 row matrix from a file, containing a 3x2 matrix:

unix> cat data.dat
1 2 3
4 5 6

IDL> data = FltArr(3,2) ; create the data array to contain the matrix
IDL> NameOfFile = 'data.dat'
IDL> openr, 5, NameOfFile ; open the file for read
IDL> readf, 5, data ; read the data into the file
IDL> close,/all ; close all open file


You can then use the REFORM statement to split the columns and rows if needed.

2. Read an ASCII file that has 10 lines of header information, followed by a 300x400 floating point array and two 300-element integer vectors. One way to do it would be:

IDL> header = StrArr(10) ; create a string array of 10 elements
IDL> array = fltarr(300,400) ; create a float array of 300x400 elements
IDL> vector 1 = IntArr(300) ; create a integer vector
IDL> vector 2 = intarr(300) ; create another integer vector
IDL> openr, lun, NameOfFile, /Get_Lun ; open file, and the /Get_Lun will run the Get_Lun pro.
IDL> readf, lun, header, array, vector1, vector2
IDL> close, lun

After reading in a line, you can use the various string functions to extract portion of the line into another variable. Examples of string functions: STRTRIM, STRMID, STRPOS, STRPUT, STR_SEP, STRCOMPRESS, STRLEN ...

IDL> print,mystring
My name is wombat.
IDL> print,strmid(mystring,3,4) ; extract the string, starting at the 3 and take 4.
name



Writing Formatted Data


There are two commands in IDL for writing formatted data
Print Writes formatted output to the standout
PrintF Writes formatted output to a file


Examples of Writing Data

1. To write the 3x2 matrix out back to a file:

IDL> print, data ; just to take a look at the data
1.00000 2.00000 3.00000
4.00000 5.00000 6.00000
IDL> openw, 33, 'DataOutput.dat' ; open the file for write
IDL> printf, 33, data ; write the data into the file
IDL> close, 33


2. To write explicit format output

IDL> openw, lun, 'datafile.dat', /get_lun
IDL> printf, lun, var1, var2, format='(3(8(f6.2,x)),10I5))
IDL> free_lun, lun


3. To write a string

IDL> MyString = "My name is wombat."
IDL> Openw, 2, 'StringOutput.dat'
IDL> printf,2, MyString ; write the string into the file
IDL> Close,2
IDL> $cat StringOutput.dat
My name is wombat.
IDL>




Reading and Writing Unformatted Data


Unformatted Data, in a sense, is simply a continuous data stream. In this case, you will use the following two commands:

ReadU Reads unformatted data from a specific file.
WriteU Writes unformatted data to a specific file.


Example:

IDL> $cat unformat.dat ; $ <- goto operating system, cat unformat.dat
00010002000300040005000600070008
IDL> var=intarr(16) ; define var to be an integer array of 16 elements
IDL> openr,1,'unformat.dat
IDL> readu,1,var read the unformatted data into the integer array
IDL> close,/all
IDL> print,var
12336 12337 12336 12338 12336 12339 12336 12340 12336
12341 12336 12342 12336 12343 12336 12344


Note: The numbers are actually ascii numbers, so you will get a non-zero byte value for 0.


Managing and Working with Data Files


Testing for End-of-File

The EOF function tests to see if the file pointer is currently positioned at the end of the file. It returns true (or 1) if the answer is yes, and false (or 0) otherwise. Below is an example code fragment.

openr, lun, 'datafile.dat', /get_lun ; open file
data = fltarr(1000) ; define data to be a float array of 1000 elements
tmp = 0.0 ; initiate the tmp variable
i = 0 ; initiate the index for the loop
while (not EOF(lun)) do begin
; Read one value at a time until EOF
readf, lun, tmp ; read from file

data(i) = tmp
i = i + 1

endwhile
data = data(0:i-1) ; trim the data at the end.


Getting Help and Information

To get information about open file units, use the Help command with the Files keyword:

IDL> Help, /Files


To get more specific information about a particular open file unit, use the FStat function.

Example:

IDL> openr,1,'unformat.dat
IDL> info=fstat(1)
IDL> print,info.size
33
IDL> print,info.cur_ptr
0


Positioning File Pointers

The Point_LUN procedure is used in IDL to position the file pointer within a data file.

To point to the beginning of a file:

IDL> Point_LUN, lun, 0


You can move back and forth between several file locations by using Point_LUN with Fstat:

IDL> info = fstat(lun) ; get the file status
IDL> current = info.cur_ptr ; store the current location
IDL> readu, lun, variable ; read the variable
IDL> point_lun, lun, current ; return to the previous stored location


STEP 3: Closing the file when you are done


Remember to use the close statement to close the file when you are done. Closing a file does two thing: the lun used by the file will be freed, and you can "recycle" the lun in the program. It also prevents any modifications to the file.

To close a file, simply use the CLOSE statement:

IDL> close, lun
or
IDL> close,/all ; this will close all the files




Paul Lee
updated: 2/5/99