Before beginning

Download directories.zip to retrive folders:

• avm_file_structure
• avm_panels
• metadata_template_structure

1. Use this Folder Layout (from avm_file_structure folder).

2. Get an Archive account. Be familiar with downloading fits file from Archive.

• http://archive.stsci.edu/registration/registration_form.htm

3. Get a Flux account and Linux account

• Group to join is "oponvo"

4. Be familiar with how to download fits from HLA.

• http://hla.stsci.edu/hla_faq.html#Images3

5. Download AVM panels on hard-drive for Mac user:

• Go http://virtualastronomy.org/avm_metadata.php
• Go to "Adobe Metadata Panels"
• Download: AVM Adobe CS4-CS5 Metadata Panels
• Move the "Astronomy" folder in Macintosh HD/ Library/Application Support/Adobe/XMP/Custom File Info Panels/3.0/panels**
• Move the "Astronomy.xml" in Macintosh HD/Library/Application Support/Adobe/XMP/Custom File Info Panels/3.0/custom** (CS5)
• Move the "Astronomy.xml" in Macintosh HD/Library/Application Support/Adobe/XMP/Custom File Info Panels/2.0/custom** (CS4)
• Note that to function properly it may be necessary to configure a SWF trust file in the location: Library/Application Support/Macromedia/FlashPlayerTrust/
  
  • At this location there should be a file (ending in .cfg, i.e. Astronomy.cfg) that contains a line of text containing the path to the appropriate directory described above.
    
  • /Library/Application Support/Adobe/XMP/Custom File Info Panels/2.0/panels/Astronomy

Note: We were able to successfully download the panels for CS4.

**Need administrator to grant permission to move files into respective directories.

6. Download Chandra's Pinpoint WCS

• Go to https://www.cfa.harvard.edu/~akapadia/pinpointwcs/
• Locate "Download" and right click on "Download for MAC OS X(Intel)"
• Open ".dmg"file and and drag "Pinpoint WCS' into Application folder

______________________________________________________________________________________________________________________________________________________________

A. Retrieve tiff images

1. Go to http://hubblesite.org/newscenter/archive/releases/and specify the year, number, and letter of release. For example, http://hubblesite.org/newscenter/archive/releases/2008/07/image/a/

2. Locate “Highest-quality download options” and then click on the "Go"

3. Download large file tiff and save into directories/avm_file_structure/Pinpoint/Tiff_files.

______________________________________________________________________________________________________________________________________________________________

B. Retrieve fits files through Hubble Legacy Archive

1. Go to http://hla.stsci.edu

2. Click on "Enter site here"

3. In search box type in object name and click on "advanced search". You are able to specify instrument(s) and Proposal ID. Then click on the "Search" button.

Object name: NGC 1132 Instrument: ACS PropID: 10558

4. Locate desired fits file, click on the cart, and then click on the yellow tab. Download fits file by clicking on "Fetch HLA Data". Place fits file in directories/avm_structure_file/Pinpoint/Fits_files.

______________________________________________________________________________________________________________________________________________________________

C. Retrieve fits files via MAST

1. Go to http://archive.stsci.edu/hst/search.php

2. Enter object name under "Target Name", enter "Proposal ID", and select Instrument (i.e. Imagers). Then click on "Search." (Side note: it is a good practice to make sure you reset the "Output Columns").

Target name: NGC 1132 Instrument: ACS PropID: 10558

3. Locate the "Dataset" and "Target Name" columns and click on either the dataset name or target name link (a new browser will open). For example, click J9BQ02020 or NGC-1132 .

4. Scroll to the bottow of the page and click on "Retrieve Data".

5. Enter "Archive Username" and "Archive Password". Select "Stage: Put the data onto the Archive staging disk*". Select "Calibrated" under "Science Files Requested" and "DRZ" for "File Extensions Requested" (if available). When you are finished making your selections click on "Send retrieval request to ST-DADS".

6. Archive will send you two emails: the first stating that your request has been recieved and the second stating you can retrieve your data via ftp using your archive account and password
to log-in. To retrieve fits file open a terminal window. Ftp to stdatu.stsci.edu. You will be prompted to enter your archive username and password. Archive will email you the name of the file and directory. For example: /stage/username/username07296.

7. To retrieve the fits files enter this commands below in a terminal window:

ftp stdatu.stsci.edu > username > password > cd stage > cd username > cd username07296 > get j9bq02020_drz.fits (filename)

8. Place fits file in avm_structure_file/Pinpoint/Fits_files

______________________________________________________________________________________________________________________________________________________________

D. Retrieve Metadata

1. Go to the Fast Facts on hubblesite.org

2. Copy and paste fast facts contents into an excel spreadsheet. In the spreadsheet you will want to copy and paste the table to the right, it will be useful later.

DatasetID: Tempororal.StartTime: Temporal.IntegrationTime: Spectral.ColorAssignment: Spectral.Bandpass Spectral.Band   Subject.Name Title: Credit: Distance.Notes: PropID: Instrument: Release Date: Spectral.Notes Constellation: Filters:

3. Go to http://archive.stsci.edu/hst/search.php and search by Target Name and PropID.

4. In “Output Format” change “HTML_Table” to “Excel_Spreadsheet”

5. Copy and paste information into excel spreadsheet. Save into directories/avm_file_structure/MAST/Pre-mod.

6. You can delete RA, DEC, Ref integer, Stop time, Apertures, Releaes Date, Preview Name, and High-Level Science Products columns. (Warning: before deleting or changing information from MAST make sure you have two excel spreadsheets saved. For example save unmodified execel as 2008-07-mast and then save modified exel as 2008-07-mast-mod. Save 2008-07-mast-mod into directories/avm_file_structure/MAST/Post-mod.)

7. Sort by StartTime and Filters/Grating. Sum the Exp time for the various filters. Then concatenate Facility, Instrument, Spectral.Wavelength, DatasetID, Temporal.StartTime, Temporal.IntegrationTime, Spectral.ColorAssignment, Spectral.Bandpass, and Spectral.Band values.

The formula for concatenation is: CELL1,";", CELL2, ";", CELL3, etc.

______________________________________________________________________________________________________________________________________________________________

E. Pinpoint

1. Before using pinpoint:

• Copy tiff images in directories/avm_file_structure/Pinpoint/Tiff_files into directories/avm_file_structure/Pinpoint/Output_tiff_files. (Once tiff images are dragged into Pinpoint WCS and they are exported with WCS metadata).
• For PinPoint WCS program drag fits files from directories/avm_file_structure/Pinpoint/Fits_files and tiff files from directories/avm_file_structure/PinpointOutput_tiff_files.
• Save exported fits files in directories/avm_file_structure/Pinpoint/Output_fits_files.
• View of before and after metadata in Bridge after using Pinpoint:

2. After pinpoint:

• Open SAOImage DS9 (For more information about DS9 go to http://hea-www.harvard.edu/RD/ds9/).
• Open fits file from HLA/MAST.
• Go to "Frame", click on "new", then click on "title".
• Open output fits file from pinpoint.

• Click on "scale" to adjust the view of fits image by using highlighted 2 options. To adjust image further go to "Scale" > "Scale Parameters..." and adjust the green line to the left.

• In the Menu bar go to "Frame" > "Match Frames" > "WCS".
• Click on "scale" and then "blink". DS9 will blink both fits files (if the WCS is a perfect match there will be no movement.)
  • Save screen snap of images in avm_file_structure/Pinpoint/Pinpoint_jpg_match

After Pinpoint: 2008-07-c.jpg	Match in DS9: 2008-07-c-match.jpg

______________________________________________________________________________________________________________________________________________________________

F. Exporting XMP files/Replacing Boilerplate on Image:

• I provide a master bp with common things such as creator and contact info in metadata_templates_structure/master_bp
• To create master/individual boilerplates:
  • Go to "Tools" > "Create Metadata Template..."
• To edit boilerplates:
  • Go to "Tools" > "Edit Metadata Template"
• To apply boilerplates:
  • Go to "Tools" > "Replace Metadata"
• To export boilerplates:
  • Go to "File" > "File Info..."
  • Shortcut: shift+alt/option+command+I
• Structure of the Metadata Template folder is shown below:

• Bridge can only see the boilerplates/XMP files when they are directly in the "Metadata Templates" folder.
  • For Mac: Macintosh HD/Library/Application Support/Adobe/XMP/Metadata Templates
• Difference in boilerplate files and export XMP files:

Boilerplate	Exported XMP

If you are having issues viewing Metadata tags in Bridge "Metadata" tab, go to the Menu bar in Bridge, select "Preferences...", select "Metadata", and make changes below:

Select:

• File Properties (Select shown below):

• IPTC Core (All)
• Astronomy Visualization Metadata (All)

Deselect:

• IPTC (IIM, legacy)
• Camera Data (EXIF)
• GPS
• Camera Raw
• Audio
• Video
• Version
• Cue
• DICOM
• Mobile SWF

______________________________________________________________________________________________________________________________________________________________

G. Metadata Gathering:

	AVM Tag	Status	Where to get metadata	Next	Format	Other names	Comment
1	Creator		Master Boilerplate	MBP
2	CreatorURL	Required	Master Boilerplate	MBP		Creator.Website(s)
3	Contact.Email		Master Boilerplate	MBP		Creator.Email	Deprecated
4	Contact.Name		Master Boilerplate	MBP		Creator	Same as AVM tag Creator
5	Contact.Telephone		Master Boilerplate	MBP		Creator.Phone(s)
6	Contact.Address		Master Boilerplate	MBP		Creator.Address
7	Contact.City		Master Boilerplate	MBP		Creator.City
8	Contact.StateProvince		Master Boilerplate	MBP		Creator.State/Province
9	Contact.PostalCode		Master Boilerplate	MBP		Crerator.PostalCode
10	Contact.Country		Master Boilerplate	MBP		Creator.Country
11	Rights	Required	Master Boilerplate	MBP		Rights Usage Terms
12	Title	Required	Script	CP
13	Headline		By hand	CP	Other		Use Fast Facts information and script information
14	Description	Required	Script	CP
15	Subject.Category	Recommended	Script	CP	SC		Add "X." in front of word Subject.Categories, i.e.  X.Galaxy > Dwarf
16	Subject.Name	Recommended	Fast Facts	CP	S	Keywords/Object Name
17	Distance	Recommended	By hand	CP	Other		Use Distance.Notes information from Fast Facts. Convert light-years to units of parsecs
18	Distance.Notes		Fast Facts	CP		Distance
19	ReferenceURL	Required	By hand	CP	F		http://hubblesite.org/newscenter/archive/releases/2008/07/image/a
20	Credit	Required	Fast Facts	CP		Provider/Image Credit	Do not include Acknowledgments
21	Date Created		Fact Facts	CP	F	Release Date	MM/DD/YYYY
22	ID	Required	By hand	CP	F		STScI-PRC-YYYY-RR-II
23	Type	Required	Master Boilerplate	MBP
24	Image.ProductQuality		Master Boilerplate	MBP
25	ProposalID		Fast Facts	CP	SC
26	PublicationID		Master Boilerplate	MBP
27	Facility	Required	Fact Facts/MAST+	CP	SC
28	Instrument		Fact Facts/MAST+	CP	SC
29	Spectral.ColorAssignment		Fact Facts/MAST+	CP	SC	Color
30	Spectral.Band	Recommended	Fact Facts/MAST+	CP	SC
31	Spectral.Bandpass		Fact Facts/MAST+	CP	SC		Gather from the filter information provided in the Fast Facts; Do not have to include "-band"
32	Spectral.CentralWavelength		Fact Facts/MAST+	CP	SC		Gather from the filter information provided in the Fast Facts;Only use numbers
33	Temporal.StartTime		MAST+	CP	SC		YYYY-MM-DD
34	Temporal.IntergrationTime		MAST+	CP	SC
35	Spectral.Notes		Fast Facts	CP		Exposure Date(s):
36	Dataset ID	MAST Required	MAST+	CP	SC
37	Spatial.CoordinateFrame		Pinpoint	Auto
38	Spatial.Equinox		Pinpoint	Auto
39	Spatial.ReferenceValue		Pinpoint	Auto
40	Spatial.ReferenceDimension		Pinpoint	Auto
41	Spatial.ReferencePixel		Pinpoint	Auto
42	Spatial.Scale		Pinpoint	Auto
43	Spatial.Rotation		Pinpoint	Auto
44	Spatial.CoordsystemProjection		Pinpoint	Auto
45	Spatial.Quality		Pinpoint	Auto
46	Spatial.Notes		Pinpoint	Auto
47	Spatial.FITSheader		Pinpoint	Auto
48	Spatial.CDMatrix		Pinpoint	Auto			Deprecated
49	Publisher		Master Boilerplate	MBP		Source	Source and Publisher are the same tag
50	PublisherID		Master Boilerplate	MBP
51	ResourceID		By hand	CP	F		hs-YYYY-RR-II-full-tif.tif
52	ResoureURL		By hand	CP	F		http://archive.stsci.edu/pub/stpr/hs-YYYY-RR-II-full-tif.tif
53	RelatedResources		By hand	CP	F		http://hubblesite.org/newscenter/archive/releases/YYYY/RR
54	MetadataDate	Required	By hand	CP	Other		MM/DD/YY, change as you edit individual boilerplate
55	MetadataVersion	Required	Master Boilerplate	MBP

Legend:

AVM Creator	Pink
AVM Content	Light turquoise
AVM Observation	Orange
AVM Coordinate	Light blue
AVM Publisher	Grey
Auto	Automatically included in header after Pinpoint
By hand	Additional calculations and using YYYY, RR, and II to create tag
CP	Copy and paste into master-info excel/Copy and paste into individual boilerplates
DD	Day
F	Format Specific
Fast Facts	Information can be copy and pasted from Fast Facts
II	Image
MAST	Information can be copy and pasted from MAST
MAST+	Gather information from MAST and then concatenate data
Master Boilerplate	Information that is the same for all images
MBP	Information provided in the xmp file master_bp.xmp
MM	Month
Other	Additional steps, see comments
RR	Release
S	Semicolon separate for multiple values
SC	Semicolon separate for multiple values, use concatenation formula
Script	Gather information with a script. (We used scripts from Frank Summers: avm_captions and avm_metadata)
YYYY	Year

Locate example_master_info.xls:

directories/avm_file_structure/MAST/

H. Scripts

1. Go to directories/avm_file_structure/MAST/Scripts

Script names:

• avm_metadata_2010_11_30.pl
• avm_captions_2010_12_03.pl

Output text filenames:

• avm_metadata_2011_03_24.txt*
  • Output: Year, Image, Release, Date, Title, Subject.Name, Subject.Category, Instrument, Distance, ProposalID, Spectral.Band, Spectral.Bandpass, Spectral.CentralWavelength, Spectral.ColorAssignment, Temporal.StartTime, Temporal.IntegrationTime, Credit
• avm_captions_2011_03_24.txt*
  • Output: Year, Image, Release, Caption(Description)
• When opening ".txt" file go to Excel > Open... >

Select "Delimited" and click "Next >"	Select "Other:", type in "%", and click "Next >"	Click on "Finish"

• Save as excel file
  • avm_metadata_2011_03_24.xls
  • avm_captions_2011_03_24.xls

*Year, month, and date reflect the last date the scripts were run.

2. How to retrieve from flux:

ssh -XY flux

cd /home/summers/chai/scripts/avm_metadata

cp avm_metadata_2010_11_30.pl* /home/username

cp avm_captions_2010_12_03.pl* /home/username

cd /home/username

Edit pearl scripts by viewing in test editor. Change the year information as shown below:

*********************************************************************************************************

Sean's script uses the following text files to gather the captions and other pieces of metadata. Metadata included in the text file: year, release, image, date, title, subject.name, subject.category, instrument, distance, propID, spectral.band, spectral.bandpass, spectral.centralwavelength, spectral.colorassignment, temporal.starttime, temporal.intergrationtime, and credit. Sean's script also grabs metadat from the fast facts on Hubblesite.

Files needed for current work-flow (retrieve from Flux):

Note: You are able to run perl script in "summer"s directory and just copy over output text files. Current txt files include 2008-2013.

File names:

avm_metadata_2013_07_05.txt

avm_captions_2013_07_05.txt

Also, need to retrieve contents of the fast_facts_files

cd /home/summers/chai/scripts/gather_fast_facts

Run perl script on gather_fast_facts_2011_06_29.pl

cp 2012*.html /home/username/fast_facts_files

cp 2013*.html /home/username/fast_facts_files

Copy from flux:

scp file names username@name_of_system.stsci.edu:/Users/Shared/

*********************************************************************************************************

3. Viewing all press release images:

a. Go to press.stsci.edu

b. ssh -XY flux > cd db (***Get from Lisa***)

You are able to to view press release image data. They are stored by year, image name, and image release

All press release images in 2008	Data for 2008/07	Listing of images release in 2008/07	Data for 2008/07/a

______________________________________________________________________________________________________________________________________________________________

I. Picture examples

2. Example of master-info:

3. Metadata template1: IPTC Core

4. Metadata template 2: AVM (Part 1)

5. Metadata template 3: AVM (Part 2)

6. Boilerplates applied to image

Last updated: 07/05/2013