Sp−1 = {Xp−1, X2p−1, . . . , Xmp−1}
Any observations in Si can be switched with another observation in Si for i = 0, . . . , p − 1. Each of the p sets have size m and for each set we have |Si|! permutations, making the total number of permutations (m!)p. If we compute the value of the test statistic for each of these permutations we can find its probability mass function.
Regardless of the method used we see that the number of permutations can be very large, es-pecially if n is large and p is small. Therefore, in many cases it is not feasible to compute all permutations and Monte Carlo methods are used. We can use Monte Carlo methods as we only need to evaluate probabilities under the null hypothesis and we can easily generate samples from the permutation distribution (Ernst, 2004). Thus, we generate a number of samples from the permutation distribution and approximate the p-value.
Lastly, it is important to note that in order for the permutations to occur without altering the period of the function, it is essential to assume that the times are equally spaced. Otherwise, observations will no longer be in the same relative positions if swapped according to the above mentioned methods.
4.2 Test Statistics
To determine the distribution of the test statistic, its value cannot remain constant under permu-tations. If it does, the test statistic is useless. We consider the test statistic presented in Chapter3 and check whether their values remains constant under permutations. The results are summarised in Table 4.1. In this table, an ‘7’ denotes that the test statistic’s value remains constant under both hypotheses and is thus, not useful. An ‘∗’ denotes that the test statistic is constant under the null hypothesis, but varies under the alternative. In Chapter5 the benefits of this result will be discussed further. Lastly, a ‘X’ denotes that the test statistic’s value does change under both hypotheses and is thus a useful test statistic.
The first five statistics in Table 4.1 are based on the periodogram. Due to the nature of the permutations, the periodogram value at the frequency corresponding to τ0 will stay constant.
This is because the DFT term corresponding to τ0 will not change under permutations. However,
CHAPTER 4. TESTING USING PERMUTATION METHODS
Test Statistic F-statistic Fisher’s G-Statistic Bartlett’s Welch’s Vanicek’s Lomb-Scargle
(Section) (3.3.2) (3.3.3) (3.3.4) (3.3.5) (3.3.6) (3.3.7)
Method 1 7 ∗ X X 7 X
Method 2 7 ∗ X X 7 X
Method 3 7 ∗ X X 7 X
Table 4.1: This table summarises which test statistics have a constant value under permutations.
Underneath each test statistic is the cross reference to the section where the test statistic is defined in Chapter3. An ‘7’ denotes that the test statistic’s value remains constant under both hypotheses and is thus, not useful. An ‘∗’ denotes that the test statistic is constant under the null hypothesis, but varies under the alternative. Lastly, a ‘X’ denotes that the test statistic’s value does change under both hypotheses and is thus useful.
the other periodogram values do change. This was further verified via simulations using the im-plemented test described in Chapter5. Due to this property, the F-statistic will remain constant under both the hypotheses as its value is proportional to the periodogram value at the frequency corresponding to τ0. On the other hand, Fisher’s G-statistic becomes very powerful because of this property. Fisher’s G-statistic is the normalised maximum of the periodogram values. Under H0, this will be the term corresponding to τ0, which stays constant. Under H1, the maximum will be the term corresponding to τ and not τ0, which means that its value will vary between permutations.
As the basis defined in Lemma 2.3.1 is orthonormal, we know that ATA is the identity ma-trix. Moreover, from Proposition 3.2.1(i) we know that P is the diagonal matrix with diagonal entries equal to σ2. Thus, the test statistic for Vanicek’s method is s(ωj) = σ2X˜2j which is pro-portional to the periodogram value corresponding to τ0 and thus it remains constant for both hypotheses.
Bartlett’s and Welch’s method are based on the periodogram, however, different values of the periodogram are averaged allowing the test statistics to change under permutations. Lastly, the Lomb-Scargle periodogram varies under both hypotheses due to the effect of the time delay and the varying weights of Xt(See Definition3.3.2). In conclusion, only Fisher’s G-statistic, Bartlett’s method, Welch’s method, and Lomb-Scargle’s periodogram are useful for testing.
The tests with these test statistics use the permuting methods to compute the values the test statistics take under the different permutations. The p-value of the test can be determined by comparing these values. For example, the p-value for the test using Fisher’s G-statistic will be determined based on the degree of variation between its values. If they are all the same value, the p-value will be close to one, if there are all different, the p-value will be close to zero.
Chapter 5
Results & Comparison of Methods
In Chapter4we saw that four test statistics can be used to test for the period of a function using permutation methods. In this chapter, we will create new tests using permutation methods and these test statistics to construct flexible, simple, and exact tests under minimal constraints. We will evaluate these tests by applying them to functions with a single frequency. These are functions where only one frequency component is contributing to the periodic nature of the function. Later in Section5.6we address functions with more than one frequency. Examples of such functions are sums of cosines and/or sines with different co-prime periods. Lastly, a new test statistic will be presented.
5.1 Relation to Chapter 3
In Chapter3theoretical descriptions of tests using the different test statistics are given. The tests using Fisher’s G-statistic and Bartlett’s method test the hypotheses
H0T : X is Gaussian white noise against
H1T : f has frequency 2πk n
where k ∈ Jn0 corresponds to the frequency that maximises the test statistic. The tests using Welch’s method and Lomb-Scargle’s periodogram test the hypotheses
H˜0T : f does not have frequency ω against
H˜1T : f has frequency ω,
for ω ∈ [−π, π]. Exact definition of these hypotheses can be found in Chapter3.
The implemented versions of these tests using permutation methods test slightly different hy-potheses;
H0I : f has period τ0
against
H1I : f does not have period τ0,
for τ0∈ Q and τ0≥ 2. In this case, H0T and ˜H0T correspond to H1I, and H1T and ˜H1T correspond to H0I due to equation3.8. To explain this in more detail we consider the tests separately.
The theoretical test using Fisher’s G-Statistic rejects H0T if the maximum peak of the periodogram is sufficiently large. In this case, we can also conclude that the function’s period is the period corresponding to the maximum peak. The implemented version of this test using permutation
CHAPTER 5. RESULTS & COMPARISON OF METHODS
methods evaluates how this maximum peak varies under permutations. If the true period is used for the permutation methods described in Chapter4, the shape of the periodogram will stay the same by definition of periodicity. Thus, the maximum value does not differ between permutations.
However, in the case that we are permuting under an incorrect period, the periodogram values will average out. Hence, the maximum value of the periodogram will decrease, with the difference distributed among the other periodogram values. Therefore, the implemented test rejects H0I if the maximum value of the periodogram decreases. Thus, the theoretical test rejects H0T if there is a peak in the periodogram, whereas the implemented test fails to reject H0I in this case. The test using Bartlett’s method is implemented analogously, however, rather than only considering the maximum peak, it considers the maximum sum of subsequent periodogram values. The period corresponding to the last summand determines the period of the function.
Although the distribution of Fisher’s G-statistic is known, the implemented test leverages the fact that under the null hypothesis Fisher’s G-statistic remains constant and only varies under permutations under the alternative hypothesis. Due to the contrasting behaviour under the two hypotheses, the test is expected to perform exceptionally well. Moreover, in Chapter3, we saw that we only had an approximate distribution for Bartlett’s method. Thus, using permutation methods will allow us to find an exact distribution, making the test more accurate.
Welch’s method defines a more accurate periodogram which we denote by I0. The theoretical test uses components of this periodogram as test statistics. Given a frequency value, the test rejects ˜H0T if the value of I0 corresponding to this frequency is large. The implemented version of this test uses the value of I0 corresponding to τ0as its test statistic. If τ0is the period of the func-tion, the permutations should not drastically alter the test statistics value. Thus, the implemented test rejects H0I only if the observed value of the test statistic is extremely different from its values for other permutations. The variation in the value of the test statistic occurs as a result of the averaging performed when permuting with an incorrect period. Thus, the theoretical test rejects H˜0T if the test statistics value is large, whereas the implemented test fails to reject H0I in this case.
Similar to the test using Bartlett’s method, this statistic only had an approximate distribution in the theoretical test. Therefore, using permutations will give us an exact distribution, making the test more accurate. This is one of the main benefits of using permutation methods. Lastly, the test using Lomb-Scargle’s periodogram is analogous to the test using Welch’s method.