APSB Technical Report Number 1993-02 Gopher It Man R. E. JACKSON DRAFT April 1993 Abstract This document describes how STEIS was Gopher'd and WAIS'ted. STEIS maintainers and other Gopher server maintainers may find this information useful. TABLE OF CONTENTS 1. Introduction 3 2. WAIS Software 3 2.1. STScI Modifications 3 3. Gopher Software 3 3.1. STScI Modifications 3 3.2. ts and tb Tools 4 4. WAIS Server 4 4.1. Maintenance 5 5. Gopher Server 5 5.1. Maintenance 6 6. Search STEIS Menus 6 7. .Links Files 7 7.1. Search Local WAIS Index 7 7.1.1. Maintenance 7 7.2. Gopher Access 7 7.3. Telnet Access 7 7.4. FTP Access 8 7.5. WAIS .src Files 8 8. .cap Subdirectory Files 8 9. *.src Files 9 10. Shell Commands 9 11. CRON Jobs 9 11.1. Maintenance 10 12. STScI Documents 10 12.1. Maintenance 10 13. Possible Enhancements 11 13.1. Finer Grained and Complete Document Files 11 13.2. Finer Grained and Reformatted AEC.catalog Files 12 Revision History: Revision A - original with each revision mention any major changes, date, your name 1. Introduction Gopher and WAIS are public domain tools providing easy access to information. They have been implemented on top of the existing STEIS system. This document describes how it was implemented and how to use the Gopher and WAIS tools on STEIS and similar in- formation repositories. 2. WAIS Software The WAIS software was obtained from: sunsite.unc.edu:pub/wais/freeWAIS0.1.tar.Z This version of WAIS is enhanced to support partial Boolean searches. These modifica- tions only affect the server portion of WAIS. 2.1. STScI Modifications The following file was obtained from wheat.tc.cornell.edu:pub/wais/* add-ps-to-wais It was modified to use the already existent ASCII text file created by DVI2TTY or other tools, rather than using the PS2ASCII tool. 3. Gopher Software The Gopher software was obtained from: boombox.micro.umn.edu:pub/gopher/Unix/gopher1.12.tar.Z This includes a LineMode client for both UNIX and VAX/VMS. The object and gopher directories were installed on STEIS at software/gopher-client to provide easy access for VAX/VMS users who cannot Uncompress and Untar files. 3.1. STScI Modifications The gopher1.12 server has the feature that the Gopher type is always set to TEXT for items retrieved via a local WAIS query. Thus the client program cannot recognize .ps Postscript files as requiring the Image viewer. A second feature was the addition of an extra line to files retrieved by local WAIS queries. These extra lines prevented the files from being processed by XDVI and GHOSTVIEW. To fix these problems, minor changes were made to: gopherd/Waisindex.c gopherd/gopherd.c In Waisindex.c, a simple and dumb test was made on the filename or the WAIS type, and all PS, DVI, GIF, and TIFF files were returned with the Gopher Type of IMAGE. In most other Gophers, PS and DVI files are usually Gopher typed as TEXT. This type con- version in Waisindex.c and in gopherd.conf could be changed so they are again typed as TEXT. In gopherd.c, the printing of the extra lines was commented out. Additionally, gopherd.c was modified to display the file modification date and file size for TEXT files. This extra information should be useful. The code changes are contained in date+size.patch. 3.2. ts and tb Tools The software needed for the "Search ALL Menus" is normally obtained from: boombox.micro.umn.edu:pub/gopher/Unix/GopherTools/tstb.tar.Z However, a new and improved version was obtained by mail from the author. The tsc.c file was compiled by GCC and renamed to tsc. The tbnet script requires that PERL be installed. 4. WAIS Server The Makefile was edited to use the CFLAGS -DBOOLEANS -DPARTIALWORD -DLITERAL and not -DBIO. This was the recommendation of the Gopher INSTALL file. The WAISINDEX and WAISSERVER executables were installed in /stsci/u1/jackson/bin but they can be installed anywhere. The server can be interactively initiated by the command: /stsci/u1/jackson/bin/waisserver -p 8000 -d /var/spool/uucppublic/.waisindex -l 0 & The -p specifies the port number. The -d specifies the directory containing the WAIS indices. Since the top of the STEIS directory tree is /var/spool/uucppublic, putting all the WAIS in- dices within the tree makes them accessible to Gopher as well as to WAIS. The -l specifies the logging level. The 0 value of the logging was chosen, since there was no obvious need to access the log- ging information. 4.1. Maintenance The file /etc/services had the following line added: z3950 210/tcp # wais server and the file /etc/inetd.conf had the following line added: z3950 stream tcp nowait root /stsci/u1/jackson/bin/waisserver waisserver -d /var/ spool/uucppublic/.waisindex -l 0 This allows WAISSERVER to run under the INET daemon, with all the attendant benefits of crash recovery and resource utilization. Whenever a new index is added, the text in the .src file should be edited to make it as ex- planatory as possible, since it is accessible via WAIS. 5. Gopher Server The Makefile.config was edited to use GCC as the compiler, since CC found fatal errors for reasons unknown. Other changes included: DOMAIN = SERVEROPTS = -DFILESTATS The SERVEROPTS value tells the compiler to include the gopherd.c feature which dis- plays the modification date and size of TEXT files. The gopherd.conf file was modified with the following lines: Admin: Bob Jackson, GopherMeister, 410-338-4737 AdminEmail: jackson@stsci.edu Site: Space Telescope Science Institute Org: Space Telescope Science Institute Loc: Baltimore, Maryland, USA ext: .ps I 9 image/PS ext: .dvi I 9 image/DVI ignore: archives ignore: lists ignore: mbox ignore: # The ext: lines tell the server to treat .ps and .dvi files as IMAGE files. This feature could be changed if client programs will automatically process .ps files through Ghostview. The ignore: lines tell the server not to display the directories archives lists mbox and to ignore any files ending in "#". The executables were installed in /stsci/u1/jackson/bin but they can be installed anywhere. The server can be interactively initiated by the command: /stsci/u1/jackson/bin/gopherd -c -u jackson /var/spool/uucppublic 8008 & Since only root can use ports lower than 1024, it was not possible to start this process at the normal port number 70. 5.1. Maintenance In order to run Gopher under the INET daemon, the following line was added to /etc/ser- vices: gopher 70/tcp # gopher server and the following line was added to /etc/inetd.conf: gopher stream tcp nowait root /stsci/u1/jackson/bin/gopherd gopherd -Ic -u jackson /var/spool/uucppublic 70 The options of -c and -u jackson appeared to be necessary in order to run the various shell scripts from within Gopher. While this may pose a security hole, the files owned by jack- son on STSCI.EDU do not contain any confidential information. 6. Search STEIS Menus A file ts containing the lines: #!/bin/sh # Version 1.2 cd `dirname $0` ./.ts/tsc $@ with a .cap directory file containing Type=7 Name=Search This Gopher Tree Numb=2 was placed in the "searchtools" directory. The .ts subdirectory must contain the files tbnet tsc One "feature" of this procedure is that it does not ignore the directories and files listed as ignore: in gopherd.conf . The .ts/.tsdata file was first generated by cd /var/spool/uucppublic /var/spool/uucppublic/.ts/tbnet stsci.edu 70 7. .Links Files Gopher provides a user transparent navigation of Internet primarily through .Links files. In .Links files, # is the comment character. The various types of links are described in the Gopherd Man page and in the examples di- rectory of the Gopher software distribution. The links used by STEIS are: 7.1. Search Local WAIS Index Name=Search STScI Documents Numb=1 Type=7 Path=7/.waisindex/stsci-docs Host=+ Port=+ The path is relative to the top of the directory tree served by the Gopher server. All WAIS indices were put in the .waisindex subdirectory. The use of a local WAIS index allows the Gopher server program to do the search and does not require that the waisserver program be running. Since Gopher can do some coarse grained access restriction, this would allow a WAIS index to be only locally accessible. The Numb= entry simply put that menu item first in the listing displayed by the client. 7.1.1. Maintenance Whenever a new local WAIS index is created for local use, a new entry is needed in the / wais-sources/.Links file. This allows Gopher users to easily get a copy of a .src file which they can then use to query the WAIS index directly. 7.2. Gopher Access Type=1 Name=Rest of GopherSpace (Univ. of Minn.) Host=gopher.micro.umn.edu Port=70 Path= The two main U.S. gophers at the University of Minnesota and University of Illinois at Ur- bana-Champaign were put in the top level .Links file. 7.3. Telnet Access Type=8 Name=NASA/IPAC Extragalactic Database - NED Host=ned.ipac.caltech.edu Port= Path=ned The Path= entry is supplied to the user as the suggested User name to log into the Telnet session with. 7.4. FTP Access Type=1 Name=IRAF Software Host=+ Port=+ Path=ftp:iraf.noao.edu@/ A path can be put after the "/" in the Path= entry and even pointing to a specific file. 7.5. WAIS .src Files Type=0 Name=stsci-docs.src Host=+ Port=+ Path=0/.waisindex/stsci-docs This link is used to create a Text file version of the WAIS .src file. This allows people to easily copy the .src files for use in their local xwais or swais. This type of link can provide alternative file and directory access for better user accessibil- ity and for ease of maintenance, i.e., only one copy of the file is present. 8. .cap Subdirectory Files If the .cap subdirectory contains a file with the same name, then the server uses the infor- mation in that file to override the menu name and the menu position of that file. The .cap files can contain one or both of the following lines. Name= Numb= Type= The Gopher Types are discussed in the GOPHERD man page and include: 0 Text File 1 Directory 2 CSO name server 4 Mac HQX file 7 Full Text Index 8 Telnet session 9 Binary file s Sound 9. *.src Files Gopher can search remote WAIS indices via files with the suffix ".src". These are the same .src files as used by Xwais. For example the STELLAR Abstracts abstracts.src file looks like: (:source :version 3 :ip-address "128.183.36.18" :ip-name "ndadsb.gsfc.nasa.gov" :tcp-port 210 :database-name "abstracts" :cost 0.00 :cost-unit :free :maintainer "stelar-info@Hypatia.gsfc.nasa.gov" :description " This database is made available by the STELAR Project, part of the ... " ) A .cap subdirectory file was created for each .src file with a Name like "Search whatever". 10. Shell Commands The ability of Gopher to run shell commands is utilized to run the EXT and EMAIL com- mands. In the case of the EXT command, there is a file "search-ext" containing the lines: #!/bin/sh echo "0Matches for $1 exec:$1:.ts/.ext stsci.edu 70\r" echo ".\r" and with a .cap subdirectory entry of: Type=7 Name=Search STScI Phonebook Numb=1 The file ".ts/.ext" contains the lines: #!/bin/sh ext $1 Both the search-ext and the .ts/.ext files must have their execute permissions set. The "70" in the search-ext file must be the port number used by the Gopher server. There are similar files for the EMAIL command. 11. CRON Jobs As soon as a file appears in the Gopher directory tree it is visible to a Gopher user. How- ever the WAIS indices and the "Search ALL Menus" .tsdata file are static and must be up- dated via CRONTAB entries. There is a file .ts/cron containing the following lines: 0 1 * * * cd /var/spool/uucppublic; /var/spool/uucppublic/.ts/tbnet stsci.edu 70 0 2 * * * cd /var/spool/uucppublic/stsci/hst_news; /stsci/u1/jackson/bin/waisindex -d /var/ spool/uucppublic/.waisindex/hst-status hst_status_* old_reports_*/* 0 3 * * * cd /var/spool/uucppublic/observer/completed_observations; /stsci/u1/jackson/bin/ waisindex -d /var/spool/uucppublic/.waisindex/hst-weekly-summary week_summ_* sum- maries_*/* 0 4 * * * cd /var/spool/uucppublic/observer/weekly_timeline; /stsci/u1/jackson/bin/waisin- dex -d /var/spool/uucppublic/.waisindex/hst-weekly-timeline timeline_* 0 5 * * * cd /var/spool/uucppublic/observer/long_range_plan; /stsci/u1/jackson/bin/waisin- dex -d /var/spool/uucppublic/.waisindex/hst-long-range-plan long_rang_plan* The first entry updates the "Search ALL Menus" file .tsdata and is run every night at 1 AM. The remaining lines update the WAIS indices for those files which are periodically updat- ed. The weekly-timeline index only includes the current years weekly timeline files. The hst- status and the hst-weekly-summary indices include the current years and all past years files. Their intervals could be changed to correspond to their normal update times. However they run quite quickly and guarantee that none of the WAIS indices are more than 24 hours out of date.. 11.1. Maintenance The CRON job sends mail containing standard output of the last four jobs. The file date of the .tsdata file can be check to see when it last ran. 12. STScI Documents Each STScI user document has been put into its own subdirectory in the documents direc- tory. The documents directory contains a .Links file with Name=Search STScI Documents Numb=1 Type=7 Path=7/.waisindex/stsci-docs Host=+ Port=+ Each subdirectory contains a ASCII version of the document with a .txt file suffix and a PostScript version of the document with a .ps file suffix. 12.1. Maintenance Assuming that each subdirectory in documents already has both ASCII (.txt) and PostScript versions of the documents, then the following procedures can be used to create and update the stsci-docs WAIS index. New documements can be added by: cd /stsci/u1/jackson/bin/waisindex -a \ -d /var/spool/uucppublic/.waisindex/stsci-docs *.txt /stsci/u1/jackson/bin/add-ps-to-wais \ /var/spool/uucppublic/.waisindex/stsci-docs *.ps If a document which is already indexed is changed, then for the first directory in docu- ments: cd /stsci/u1/jackson/bin/waisindex \ -d /var/spool/uucppublic/.waisindex/stsci-docs *.txt /stsci/u1/jackson/bin/add-ps-to-wais \ /var/spool/uucppublic/.waisindex/stsci-docs *.ps and then do: cd /stsci/u1/jackson/bin/waisindex -a \ -d /var/spool/uucppublic/.waisindex/stsci-docs *.txt /stsci/u1/jackson/bin/add-ps-to-wais \ /var/spool/uucppublic/.waisindex/stsci-docs *.ps for the remaining directories. It is possible to create the WAIS index for all documents by executing the shell inscript ".ts/ index-documents". As new directories are added, this shell script should be updated. The ASCII (.txt) files were created by the following means: TeX and LaTeX dvi2tty -w 132 -o .txt .dvi Framemaker Save As Text .txt Word Save As Text .txt For FOS Instrument Handbook figures, the caption was entered as the .txt file. For FOS Handbook tables, the Framemaker Save As Text output was used for the .txt file. Ideally, each document should be delivered as a set of PostScript and ASCII files, one for each section or chapter. Each file should contain ALL the figures and tables for that section or chapter, i.e., no by hand pasteup. 13. Possible Enhancements There are a number of desirable enhancements to the STEIS Gopher. 13.1. Finer Grained and Complete Document Files Several of the Phase I Documents were single large TeX or LaTeX files. If they were de- livered as mulitple smaller files, then the WAIS index would return only those parts of the document with the appropriate information. This would be very useful for the PostScript versions since the Ghostview viewer cannot search for a specific string. The FOS Instrument Handbook, and possibly others, needs to have the Figures and Tables integrated with the text, so that the PostScript file contains the entire document. Both of these require some work on the part of the document authors. 13.2. Finer Grained and Reformatted AEC.catalog Files The Archive Exposure Catalog is 16 Megabytes and requires the user to download it in or- der to find if their information is present in it. If it were broken up into several files, e.g., by month, and the files put into a WAIS index, the catalog could be searched using WAIS. With minor changes to the RA and DEC values, e.g., replacing 01 02 03.4 with 01H02M03.3S, the partial string matching ability of WAIS could provide a crude area searching ability. The AEC file providers have been requested to make these changes.