Beginner's IDL - Data I/O

Beginner's IDL - Data Input and Output

The objective of this chapter is to provide a more in-depth look at a specific topic in IDL. Here we will look at how to read and write data from IDL.

Brief Recap from Beginner's IDL I
IDL can read many different file formats
To change directory
Reading and writing files in general
Reading Formatted Data
Examples of Reading Data
Writing Formatted Data
Reading and Writing Unformatted Data
Managing and Working with Data Files
Closing the file when you are done

Brief Recap from Beginner's IDL I

Before diving into data input and output in IDL, consider the methods we discussed before.

How to read and write files

Read data

How to read in a table with 2 columns, 5 rows

Method 1:
IDL> data=fltarr(2,5) ; create the data array
IDL> openr,lun,'in_array.dat',/get_lun ; open the data file
IDL> readf,lun,data ; read data from the file
IDL> close,/all ; close the data file
IDL> help,data ; check the data array
DATA FLOAT = Array[2, 5]
IDL> print,data
1.00000 5.00000
2.00000 6.00000
3.00000 7.00000
4.00000 8.00000
5.00000 9.00000
IDL> x=reform(data(0,*)) ; separate the columns
IDL> y=reform(data(1,*))
IDL> print,x
1.00000 2.00000 3.00000 4.00000 5.00000
IDL> print,y
5.00000 6.00000 7.00000 8.00000 9.00000

Method 2:
IDL> x=fltarr(5) ; create the x data array
IDL> y=fltarr(5) ; create the y data array
IDL> openr,lun,'in_array.dat',/get_lun ; open the data file
IDL> .run ; execute a group of control statement
- for j=0,4 do begin ;start of the loop
- readf,lun,tmp1,tmp2 ; read the j-th row
- x(j)=tmp1 ; put the data into the x array
- y(j)=tmp2 ; put the data into the y array
- endfor ; end of the for loop
- end ; end of the control group
% Compiled module: $MAIN$.
IDL> print,x
1.00000 2.00000 3.00000 4.00000 5.00000
IDL> print,y
5.00000 6.00000 7.00000 8.00000 9.00000

How to read in unformatted binary data

IDL> openr, lun, 'galaxy.dat', /get_lun ; open the data file
IDL> image=bytarr(256,256) ; create a 256x256 byte array
IDL> readu, lun, image ; read in the data
IDL> free_lun, lun ; close the file

How to read in HST data

Check out the widget_olh

Write data

Replace openr (open-read) with openw (open-write), and replace readf (read-file) with printf (print-file).

How to write the 2 by 5 array back:

IDL> openw,lun,'out_array.dat',/get_lun
IDL> printf, lun, data
IDL> Free_lun, lun

or

IDL> openw,lun,'out_array.dat',/get_lun
IDL> printf, lun, x, y
IDL> Free_lun, lun

We will take a closer look at reading and writing data in IDL here.

IDL can read many different file formats

As the heading suggests, IDL can read and write in many different file formats. There are IDL library routines that are readily available for use in reading some common formats, such as PICT, JPEG, TIFF, etc. Also, there are other routines written by others that would allow you to read FITS file as well as IRAF images. Table 1 here summarizes the various file formats and the corresponding IDL routines to use for reading the image data.

File Format Routine to Read Routine to Write

BMP Read_BMP Write_BMP

GIF Read_GIF Write_GIF

Interfile (v 3.3) Read_Interfile None

JPEG Read_JPEG Write_JPEG

PICT Read_PICT Write_PICT

PGM/PPM Read_PPM Write_PPM

PostScript None Set PS Graphics Device

Sun Rasterfiles Read_SRF Write_SRF

SYLK Read_SYLK Write_SYLK

TIFF TIFF_Read TIFF_Write

Wavefront Advanced Visualizer (WAVE) Read_WAVE Write_WAVE

X11-Bitmap Read_X11_Bitmap None

XWD Read_XWD None

CDF see CDF library see CDF library

netCDF see netCDF library see netCDF library

HDF See HDF library see HDF library

Table 1. File formats and routines to read and write them.

To change directory

IDL> cd,'/data/wombat1/IDL_course'

Reading and writing files in general

There are three steps in reading and writing files:

Open a file for read, write, or update.
Read, write, or update the content of a file
Close the file

Step 1: Open a file for read, write, or update

There are three commands for opening a file:

OpenR Open a file for reading

OpenW Open a file for writing

OpenU Open a file for updating

The form of these commands:

IDL> OpenR, lun, 'filename'
or
IDL> OpenW, lun, 'filename'
or
IDL> OpenU, lun, 'filename'

where lun is a logical unit number and filename is a case sensitive name of the file you would like to open. You can also use a string variable instead.

IDL> NameOfFile = 'datafile.dat'
IDL> OpenR, lun, NameOfFile

About the Logical Unit Number (lun)

The Logical Unit Number (lun) is basically an index number assigned to an open file, and you can use this index number to refer to the file. In a way, it is like a pointer. The Open statement is to associate a filename with a lun. Three luns are reserved by the system and not available to the user:

0 The standard input stream

-1 The standard output stream

-2 The standard error stream

There are 128 other logical unit numbers that you can use:

1-99 You can use directly, but you will have to keep track of them yourself

100-128 You can use through the Get_LUN and Free_LUN procedures.

To use the lun between 1 and 99:

IDL> OpenR, 25, NameOfFile

Once a lun is assigned to a file by you, it cannot be reassigned until the file is closed by you. To close a file, simply use the CLOSE statememt:

IDL> Close, 25

To use the lun between 100-128, you will have to use the Get_LUN and Free_LUN procedures. The advantage of using the two procedures is that you will not need to keep track of the usage of the luns. As long as you do not use up all the luns, you will be guaranteed a valid lun return by the procedure. To open and close a file:

IDL> Get_LUN, lun
IDL> OpenR, lun, NameOfFile
IDL> ... do something...
IDL> Free_LUN, lun

STEP 2:

Reading Formatted Data

IDL can read files of with the following type of data formatting:

Free Format A free format file uses either commas or whitespace (tabs and spaces) to distinguish each element in the file

Explicit Format An explicit format file is formatted according to tules specified in a format statement. The IDL format statement is similar to the kind of format statement you would write in a FORTRAN program.

There are three commands for reading free formatted data:

Read Reads free format input from standard input, usually the keyboard.

ReadF Reads free format input from a file.

ReadS Reads free format input from a string variable.

Rules for reading free format data:

Input data must be separated by commas or whitespace.
Input is performed on scalar variables. Arrays and structures are treated as collections of scalar variables.
If the current input line is empty, and there are still variables left requiring input, read another line.
If the current input line is not empty, but there are no variables left requiring input, ignore the remainder of the line.
Convert data into the data type expected by the variable.
If reading into a string variable, all characters remaining on the current line are read into the variable.

So to read free format data:

IDL> array = IntArr(5)
IDL> Read, array

To read explicit format data:

IDL> ReadF, lun, var1, var2, Format='(3(8(F6.2,X)), 10I5)'

Refer to the IDL Reference for the format specifiers.

Examples of Reading Data

1. To read in a 3 column by 2 row matrix from a file, containing a 3x2 matrix:

unix> cat data.dat
1 2 3
4 5 6

IDL> data = FltArr(3,2) ; create the data array to contain the matrix
IDL> NameOfFile = 'data.dat'
IDL> openr, 5, NameOfFile ; open the file for read
IDL> readf, 5, data ; read the data into the file
IDL> close,/all ; close all open file

You can then use the REFORM statement to split the columns and rows if needed.

2. Read an ASCII file that has 10 lines of header information, followed by a 300x400 floating point array and two 300-element integer vectors. One way to do it would be:

IDL> header = StrArr(10) ; create a string array of 10 elements
IDL> array = fltarr(300,400) ; create a float array of 300x400 elements
IDL> vector 1 = IntArr(300) ; create a integer vector
IDL> vector 2 = intarr(300) ; create another integer vector
IDL> openr, lun, NameOfFile, /Get_Lun ; open file, and the /Get_Lun will run the Get_Lun pro.
IDL> readf, lun, header, array, vector1, vector2
IDL> close, lun

After reading in a line, you can use the various string functions to extract portion of the line into another variable. Examples of string functions: STRTRIM, STRMID, STRPOS, STRPUT, STR_SEP, STRCOMPRESS, STRLEN ...

IDL> print,mystring
My name is wombat.
IDL> print,strmid(mystring,3,4) ; extract the string, starting at the 3 and take 4.
name

Writing Formatted Data

There are two commands in IDL for writing formatted data

Print Writes formatted output to the standout

PrintF Writes formatted output to a file

Examples of Writing Data

1. To write the 3x2 matrix out back to a file:

IDL> print, data ; just to take a look at the data
1.00000 2.00000 3.00000
4.00000 5.00000 6.00000
IDL> openw, 33, 'DataOutput.dat' ; open the file for write
IDL> printf, 33, data ; write the data into the file
IDL> close, 33

2. To write explicit format output

IDL> openw, lun, 'datafile.dat', /get_lun
IDL> printf, lun, var1, var2, format='(3(8(f6.2,x)),10I5))
IDL> free_lun, lun

3. To write a string

IDL> MyString = "My name is wombat."
IDL> Openw, 2, 'StringOutput.dat'
IDL> printf,2, MyString ; write the string into the file
IDL> Close,2
IDL> $cat StringOutput.dat
My name is wombat.
IDL>

Reading and Writing Unformatted Data

Unformatted Data, in a sense, is simply a continuous data stream. In this case, you will use the following two commands:

ReadU Reads unformatted data from a specific file.

WriteU Writes unformatted data to a specific file.

Example:

IDL> $cat unformat.dat ; $ <- goto operating system, cat unformat.dat
00010002000300040005000600070008
IDL> var=intarr(16) ; define var to be an integer array of 16 elements
IDL> openr,1,'unformat.dat
IDL> readu,1,var read the unformatted data into the integer array
IDL> close,/all
IDL> print,var
12336 12337 12336 12338 12336 12339 12336 12340 12336
12341 12336 12342 12336 12343 12336 12344

Note: The numbers are actually ascii numbers, so you will get a non-zero byte value for 0.

Managing and Working with Data Files

Testing for End-of-File

The EOF function tests to see if the file pointer is currently positioned at the end of the file. It returns true (or 1) if the answer is yes, and false (or 0) otherwise. Below is an example code fragment.

openr, lun, 'datafile.dat', /get_lun ; open file
data = fltarr(1000) ; define data to be a float array of 1000 elements
tmp = 0.0 ; initiate the tmp variable
i = 0 ; initiate the index for the loop
while (not EOF(lun)) do begin
; Read one value at a time until EOF
readf, lun, tmp ; read from file

data(i) = tmp
i = i + 1

endwhile
data = data(0:i-1) ; trim the data at the end.

Getting Help and Information

To get information about open file units, use the Help command with the Files keyword:

IDL> Help, /Files

To get more specific information about a particular open file unit, use the FStat function.

Example:

IDL> openr,1,'unformat.dat
IDL> info=fstat(1)
IDL> print,info.size
33
IDL> print,info.cur_ptr
0

Positioning File Pointers

The Point_LUN procedure is used in IDL to position the file pointer within a data file.

To point to the beginning of a file:

IDL> Point_LUN, lun, 0

You can move back and forth between several file locations by using Point_LUN with Fstat:

IDL> info = fstat(lun) ; get the file status
IDL> current = info.cur_ptr ; store the current location
IDL> readu, lun, variable ; read the variable
IDL> point_lun, lun, current ; return to the previous stored location

STEP 3: Closing the file when you are done

Remember to use the close statement to close the file when you are done. Closing a file does two thing: the lun used by the file will be freed, and you can "recycle" the lun in the program. It also prevents any modifications to the file.

To close a file, simply use the CLOSE statement:

IDL> close, lun
or
IDL> close,/all ; this will close all the files

Paul Lee
updated: 2/5/99

File Format	Routine to Read	Routine to Write
BMP	Read_BMP	Write_BMP
GIF	Read_GIF	Write_GIF
Interfile (v 3.3)	Read_Interfile	None
JPEG	Read_JPEG	Write_JPEG
PICT	Read_PICT	Write_PICT
PGM/PPM	Read_PPM	Write_PPM
PostScript	None	Set PS Graphics Device
Sun Rasterfiles	Read_SRF	Write_SRF
SYLK	Read_SYLK	Write_SYLK
TIFF	TIFF_Read	TIFF_Write
Wavefront Advanced Visualizer (WAVE)	Read_WAVE	Write_WAVE
X11-Bitmap	Read_X11_Bitmap	None
XWD	Read_XWD	None
CDF	see CDF library	see CDF library
netCDF	see netCDF library	see netCDF library
HDF	See HDF library	see HDF library

OpenR	Open a file for reading
OpenW	Open a file for writing
OpenU	Open a file for updating

0	The standard input stream
-1	The standard output stream
-2	The standard error stream

1-99	You can use directly, but you will have to keep track of them yourself
100-128	You can use through the Get_LUN and Free_LUN procedures.

Free Format	A free format file uses either commas or whitespace (tabs and spaces) to distinguish each element in the file
Explicit Format	An explicit format file is formatted according to tules specified in a format statement. The IDL format statement is similar to the kind of format statement you would write in a FORTRAN program.

Read	Reads free format input from standard input, usually the keyboard.
ReadF	Reads free format input from a file.
ReadS	Reads free format input from a string variable.

Print	Writes formatted output to the standout
PrintF	Writes formatted output to a file

ReadU	Reads unformatted data from a specific file.
WriteU	Writes unformatted data to a specific file.