[Top] [Prev] [Next] [Bottom]

42.9 Data Echo Problem

There is a significant undiagnosed problem that can occur in time series datasets. On two different occasions, autocorrelation analyses revealed an unexpected spike of power at lags of 8.6 ms and 9.4 ms, respectively. This turned out to be caused by small patches of time series data being copied from one part of the dataset to another.

The data were organized in packets inside the HSP, and were treated as packets all the way through the system until the HST ground system at STScI reformatted them into a simple time series. The HSP team discovered that data from the beginning of packet 7 were appearing at the beginning of packet 8, and similarly for packet pairs 15 and 16, 23 and 24, and so on. The packet collection times in these two isolated instances were 8.6 ms and 9.4 ms, accounting for the results in the autocorrelation. The length of the duplicated stretch varied randomly from 12 to 24 samples.

Extensive analyses were performed by the HSP team and by experts in the operation of the Science Data Formatter (SDF), the packet interface to which the HSP sends data. No likely suspects were found. A simple "numerological" analysis also failed to implicate or exonerate any specific subsystem. For example, the HSP maintains 8 internal packet buffers, easily raising eyebrows with the period-of-8 repetition reclamation error affecting pairs of packets. That is, packets 0 and 8 shared buffers, not 7 and 8. On the other hand, packets appeared pairwise in the SDF, which maintained two ping-pong packet buffers. The SDF could generate defective packet time stamps, which then caused the ground system to put packets in the wrong order, although this was not expected to change the contents of a packet.

This autocorrelation test was performed on many datasets taken with precisely the same instrument configuration, with no irregularities found. Only four datasets (taken on 2 days less than 1 week apart) have been discovered to show this effect. If you have data taken at sample rates exceeding 1 kHz, do an autocorrelation and look for spurious power at lags equal to a packet collection time, which is the sample time multiplied by the number of samples per packet. The latter value is given in the SMS command load or in the FITS headers for the data.

[Top] [Prev] [Next] [Bottom]

Copyright © 1997, Association of Universities for Research in Astronomy. All rights reserved. Last updated: 01/14/98 16:01:49