Crop Science 42:385-392 (2002)
© 2002 Crop Science Society of America
CROP BREEDING, GENETICS & CYTOLOGY
On-Farm Strip Trials vs. Replicated Performance Trials for Cultivar Evaluation
Weikai Yan*,a,
L. A. Hunta,
Peter Johnsonb,
Gregory Stewartc and
Xuewen Lud
a Dep. of Plant Agriculture, Univ. of Guelph, Guelph, ON, N1G 2W1, Canada
b Cereal Lead, Ontario Ministry of Agriculture, Food, and Rural Affairs, Guelph, ON, N1G 5C9, Canada
c Corn Lead, Ontario Ministry of Agriculture, Food, and Rural Affairs, Dep. of Plant Agriculture, Univ. of Guelph, Guelph, ON, N1G 2W1, Canada
d Food Res. Program, Agriculture and Agri-Food Canada, Guelph, ON, N1G 5C9, Canada
* Corresponding author (wyan{at}uoguelph.ca)
 |
ABSTRACT
|
|---|
A systematic comparison between two cultivar evaluation and recommendation systems, i.e., the balanced and replicated performance trials conducted in small plots at a small number of locations, and the unbalanced and non-replicated on-farm trials conducted in large strips on many farms, is lacking. This study was initiated to investigate the usefulness of the two contrasting systems in cultivar evaluation and the relationships between them. Yield data from Ontario winter wheat (Triticum aestivum L.) strip trials and performance trials for 1998 to 2000 were analyzed by mixed models. For all 3 yr, results from the two systems were highly correlated, both in terms of the best linear unbiased predictors (BLUP) and for the t-values of BLUP. Cultivars judged to be superior (or inferior) by one system were never judged to be inferior (or superior) by the other. Thus, both on-farm strip trials and replicated small-plot trials provide valid data for effective cultivar evaluation. On the basis of t-statistics, which measure cultivar reliability, cultivars can be classified into superior (t 
2), inferior (t 
-2), and intermediate or inadequately tested (-2 < t < 2). Two cultivars can be regarded as different in reliability if their t-values differ by 3. The evaluation power of strip trials for a cultivar depends on the number of trials in which the cultivar is tested; a cultivar may not be adequately evaluated if it is tested in fewer than 20 trials.
Abbreviations: BLUP, best linear unbiased predictor
|
INTRODUCTION
|
|---|
CROP CULTIVARS are routinely evaluated in regional performance trials organized by officially recognized institutions or organizations, with data from such trials being used as the basis for cultivar recommendation. The performance trial system has been regarded as the most authentic system for making cultivar recommendations because it consists of orthogonal tests of the same set of cultivars at carefully selected locations, and is managed by trained personnel according to established protocols. It provides balanced and replicated data, which meets the requirements for data analysis by conventional statistical techniques. There has been uncertainty, however, whether small plots on research stations are sufficiently representative of large fields under on-farm conditions.
Parallel to the performance test system, an on-farm strip trial system has been developing over the past few years. Strip trials in Ontario, Canada, are organized by extension workers, seed companies, the Seed Grower's Association, and/or by farmers themselves to provide head to head comparisons among cultivars or promising breeding lines. In strip trials, entries are usually grown in non-replicated strips that are hundreds of times larger than performance trial plots. Each year, several hundred on-farm trials for winter wheat and over a thousand for maize (Zea mays L.) are conducted in Ontario. Only a few cultivars (generally two, often <5) are evaluated in a trial, however, and very often, different cultivars are tested in different trials, resulting in highly unbalanced data. Consequently, utilization of strip-trial data has been limited to paired comparisons, such as comparison between a new cultivar and a standard check (Eskridge, 1996), and only data from trials that included common cultivars can be used in such comparisons. Strip trials have not been used to make comparisons among all cultivars.
Thus, each of the two test systems, the performance trials and the strip trials, has pros and cons. The former generates balanced and replicated data from small-plots in a limited number of locations (<10 for winter wheat in Ontario), whereas the latter generates highly unbalanced and non-replicated data from large strips on numerous farms. Understanding the relationship between the two systems could have a significant impact on cultivar evaluation strategy. If they are highly correlated, either system would be sufficient; if they are complementary, both systems would be helpful; and if they are mutually exclusive, a decision must be made on which system is appropriate for cultivar evaluation. Unfortunately, this important issue has rarely been addressed, mainly because of the lack of appropriate statistical models and the limitation of computing power. These barriers are now largely removed with the advent of faster computers and advanced statistical software, such as the REML algorithm and the MIXED procedure in SAS (Littell et al., 1996; Searle et al., 1992). With these new developments in statistics and computer programs, we are now able to estimate the variance components and specific random effects or linear functions of random effects of cultivars. These effects are referred to as best linear unbiased predictor or BLUP. Piepho (1994) and DeLacy et al. (1996), among others, pointed out that the shrinkage property of BLUP is particularly useful for cultivar evaluation based on unbalanced data. Armed with this new analytical tool, and using data from winter wheat trials in Ontario, this project was initiated (i) to study the power of, and the relationships between, the strip-trial and the performance-trial systems for cultivar evaluation and recommendation, and (ii) to determine whether data from the small plots of performance trials can equate well with data obtained from farmers' fields obtained by means of farm scale equipment, and whether the highly unbalanced and non-replicated strip trials can be used for comparison among all tested cultivars.
|
MATERIALS AND METHODS
|
|---|
Strip Trial Data
Grain yield data from strip trials conducted by Ontario winter wheat growers during 1998 to 2000 were provided by the Ontario Ministry of Agriculture, Food, and Rural Affairs. A total of 149 strip trials involving 30 cultivars, 292 strip trials involving 23 cultivars, and 478 strip trials for 31 cultivars, were conducted by farmers in central and southwestern Ontario in 1998, 1999, and 2000, respectively. About 40% of the trials tested only two cultivars, and 64 to 77% of the trials tested four or fewer cultivars. Cultivars tested in different strip trials occasionally were the same, but were often different. Each cultivar was tested in from 1 to 200 trials in 1998, from 1 to 192 trials in 1999, and from 1 to >300 trials in 2000. The strips ranged from 1400 to 8000 m2. Upon maturity, whole strips were combined and weighed with a weigh-wagon, and yield was expressed at 140 g kg-1 grain moisture.
Performance Trial Data
Replicated yield data from the 1998 to 2000 Ontario winter wheat performance trials were collected from the performance trial participants. In 1998 to 2000, 31 to 34 cultivars, depending on year, were tested at eight locations within Central and southwestern Ontario. A randomized complete-block design was used at each location with four to six replications. The size of harvested plots ranged from 3.0 to 3.5 m2.
Statistical Analysis
The SAS procedure ‘MIXED’ (Littell et al., 1996) was used in analyzing both the strip-trial data and the performance-trial data. The MIXED procedure has some important advantages over the once popular GLM (i.e., general linear models) procedure. The GLM deals with only fixed effects, whereas MIXED deals with both fixed and random effects; it is, therefore, in the real-sense, generalized linear model. This is important since, in reality, many effects should be treated as random rather than fixed. The GLM produces ordinary least squares solutions of fixed effects, called best linear unbiased estimates or BLUE, whereas MIXED produces linear combinations of both fixed and random effects, called best linear unbiased predictions or BLUP. Secondly, GLM provides correct variance-component estimates only for balanced data when there are no random effects, whereas MIXED provides correct variance-component estimates in all cases because it utilizes both intra- and inter-block information. Thirdly, MIXED estimates random effects (Ri) using the restricted maximum likelihood (REML) method, which has a shrinkage relationship with the corresponding fixed effects estimated by using ordinary least-squares methods (
i):
 |
where 
2g is genetic variance, 2w represents variance-component of all other random effects, and ni is the number of trials in which cultivar i is tested. Thus, compared with i, Ri shrinks towards 0 (i.e., the grand mean) by a factor of Hi, which assumes the same formula structure as heritability (DeLacy et al., 1996) and was referred to as repeatability across trials (Piepho, 1994). Clearly, for a given 2g, the shrinkage is more severe when the experimental error, 2w, is large relative to the genotypic variance, 2g. For given 2w and 2g, the shrinkage is larger for cultivars that are tested in fewer trials (with smaller ni) than for those tested in more trials (greater ni). Consequently, the test based on random effects is more conservative for less tested new cultivars and more powerful for more extensively tested older cultivars. This shrinkage property was considered as an ideal property for cultivar evaluation (Piepho, 1994).
In this analysis, all effects were regarded as random except the intercept, and data from different years were analyzed separately. For the strip trial data, the linear model is
where Yij is the yield of genotype i in strip trial j, µ is the grand mean, gi is the random effect of genotype i, tj is the random effect of strip trial j, and eij is the error associated with genotype i in strip trial j.
For the performance trial data, the linear model is
where Yijk is the yield of genotype i in block k of location j, µ is the grand mean, gi is the random effect of the genotype i, tj is the random effect of location j, bk(j) is the random effect of block k in location j, and eijk is the error term. The degrees of freedom for the BLUP for both models were computed using Satterthwaite's procedure to ensure an appropriate t-test.
No cultivar x trial or cultivar x location term was included in the above models for two reasons. First, since the strip trials were highly unbalanced and non-replicated, it was not possible to estimate the cultivar x trial interaction. Second, the farms in the strip trials and the test locations in the performance trials were all from central and southwestern Ontario, which belong to a single mega-environment (Yan, 1999; Yan et al., 2000). A mega-environment for a crop may be defined as a portion of the crop's growing area in which there is no predictable (i.e., exploitable) genotype x location interaction. Furthermore, preliminary analysis revealed that the cultivar x location interaction variance estimated from the replicated performance trial data (0.113, 0.066, and 0.094 for 1998, 1999, and 2000, respectively) was smaller than the error variance (0.125, 0.130, and 0.241, respectively), supporting the single mega-environment assumption. As a result, pooling of the cultivar x location interaction into the experimental error decreased residual variance and prediction errors and increased experimental power.
|
RESULTS
|
|---|
Random effects for cultivars were the focus of analysis. The SAS output for the random effects of cultivars is exemplified using data from the 2000 strip trials (Table 1) and performance trials (Table 2). It includes (i) estimates for the random effect of each cultivar, which is referred to as BLUP relative to the grand mean, (ii) the prediction error of BLUP (SE), (iii) the Satterthwaite's procedure-estimated degrees of freedom (DF), (iv) the t-value for BLUP (= BLUP/SE), and (v) probability associated with each t-value.
View larger version (27K):
[in this window]
[in a new window]
|
Fig. 1. Best linear unbiased predictors (BLUP) of cultivars based on strip trials vs. performance trials. (A) 1998, (B) 1999, and (C) 2000. Cultivars 2510, 2518, 2526, 2533, 2537, 2540, 2549, 2557, 2560, and 2737 are from Pioneer Hi-Bred International Inc. Complete names for other cultivars are: ARI = OAC Ariss, ASH = Ashland, CAL = Caledonia, CAR = AC Cartier, CM67 = CM951067, CM959 = CM95009, CM96 = CM96097, DEX = AC Dexter, DLT = AC Delta, ESS = AC Essex, FRE = Freedom, FUN = Fundulea, FWB = FWB728, HAN = Hanover, HAR = Harus, HUR = Huron, KAR = Karena, LJH = LJH95, MAC = AC McKinnon, MAR = Marilee, MAX = Maxim, MEN = Mendon, MON = OAC Montrose, MOR = AC Morley, MOU = AC Mountain, MWH = MWH95-069531, PAT = Patriot, PLA = Platinum, ZOR = AC Zorro, RC98 = RC98109, RON = AC Ron, S93 = S93-1, SUP = Superior, TW95 = TW95412, TW96 = TW96273, TW97 = TW97, WIS = Wisdom.
|
|
Cultivar BLUP from Strip Trials vs. Performance Trials
The BLUP values from the performance trials were highly correlated with those from the strip trials (r = 0.64, 0.67, and 0.72 for 1998, 1999, and 2000, respectively, Fig. 1)
. With only a few exceptions, cultivars with above-average BLUP from one system tended to have above-average BLUP from the other system. These results suggest that both test systems provide valid information on cultivar evaluation, and that the two test systems are mutually valid. Figure 1 also suggests that the strip trials appeared to be more discriminating than the performance trials as evidenced by the range of BLUP values. This is particularly obvious in 1999, where the BLUP values varied from -0.65 to +0.6 Mg ha-1 for the strip trials, but they ranged only from -0.3 to +0.3 Mg ha-1 for the performance trials. Fig. 1 also reveals that the discriminating ability of the yield trials varied with years, as the range of BLUP values in 1998 (-0.9 to +0.9 Mg ha-1) was more than twice as large as that in 2000 (-0.4 to +0.3 Mg ha-1).
The t-Statistics of BLUP as a Measure of Cultivar Reliability
The t-statistic of BLUP tests whether the BLUP of a cultivar is significantly higher or lower than the grand mean, which is indicated by the associated probability (Tables 1 and 2). The associated probability for each cultivar can thus be regarded as a measure of certainty or reliability with which that cultivar would yield below or above the grand mean. Although the probability does not tell whether a cultivar tends to yield above or below the grand mean, the t-value does. Furthermore, the t-value provides a simplistic criterion for cultivar evaluation: the critical t-value at the 0.05 significance level is 2.0 when the associated degrees of freedom are 60. Therefore, roughly, cultivars with t 2 would yield significantly above the grand mean (i.e., BLUP >0) at 0.05 level; cultivars with t -2 would yield significantly below the grand mean (i.e., BLUP < 0) at 0.05 level; all other cultivars would give yields similar to the grand mean. These statements can be clearly verified from Tables 1 and 2, where cultivars are ranked by their t-values. The greater the t-value of a cultivar, the greater the likelihood that it will yield higher than the grand mean; the lower the t-value of a cultivar, the greater the likelihood that it will yield lower than the grand mean. Therefore, the t-statistic can be used as an intuitive measure of performance reliability of cultivars.
The t-Values of Cultivars from Strip Trials vs. Performance Trials
As for the BLUP values of cultivars, the t-values obtained from the strip trials and those from the performance trials were highly correlated (r = 0.68, 0.66, and 0.67 for 1998, 1999, and 2000, respectively), which further indicated that both test systems provided valid evaluation of the cultivars. This conclusion is strengthened when the cultivars are divided into superior (t 2), intermediate (2 t -2), and inferior (t -2) groups on the basis of their t-values (Fig. 2)
. Among the 18 cultivars tested in both test systems in 1998 (Fig. 2A), four (2540, 2557, 2526, and Mendon) were judged to be superior, and six (Karena, Huron, Ron, Freedom, Cartier, and Fundulea) to be intermediate by both test systems. There were interactions between the two systems, however. Cultivar 2533 was judged to be superior from the strip trials but intermediate from the performance trials; the opposite was true for cultivar 2737. Cultivars Marilee, Harus, Ariss, and 2510 were judged to be inferior from the strip trials but intermediate from the performance trials; and the opposite was true for cultivars Hanover and Montrose. Nevertheless, a cultivar that was judged to be superior from one system was never found to be inferior in the other system; equally, a cultivar that was judged to be inferior from one system was never found to be superior in the other system. This statement holds also for 1999 and 2000 (Fig. 2B and 2C).
View larger version (29K):
[in this window]
[in a new window]
|
Fig. 2. t-Values of cultivars based on strip trials vs. performance trials. (A) 1998, 1999, and (C) 2000 (see Fig. 1 for complete cultivar names).
|
|
Results presented in Fig. 2 are strong evidence that both test systems provide valid data for cultivar evaluation and recommendation, and that either system should be sufficient to identify superior cultivars for recommendation and inferior cultivars for discarding. Furthermore, since results from the two systems are partially complementary, results from both systems would lead to a greater power. Taking Fig. 2A as an example, five cultivars were judged to be superior by either system; when both systems are considered, six cultivars were judged to be superior. Similarly, two cultivars were judged to be inferior by the performance trials, and four by the strip trials. Together, six were judged to be inferior cultivars.
Predictive Power of the Two Test Systems
Cultivar trials are conducted to provide a basis for cultivar recommendation for the next year. Therefore, an important criterion for judging the validity of a cultivar evaluation system would be its predictive power for the next year's cultivar performance. For both the performance trials (Fig. 3)
and the strip trials (Fig. 4)
, t-values obtained in one year were highly correlated with those obtained in the next year, indicating that both test systems provide valid cultivar evaluation with similar predictive power.
Whether based on performance trials (Fig. 3) or strip trials (Fig. 4), a cultivar that was judged superior in one year (t 2) was never inferior in the following year (t -2); a cultivar that was inferior in one year was never superior in the following year. When the analysis was extended to data from the recent years of performance trials, it was found that although the correlations of t-values between successive years in 1989 to 1997 were lower than the correlations reported here for 1998 to 2000, the superior vs. inferior grouping of a cultivar was rarely reversed in successive years (unpublished results). Thus, cultivar groupings based on t-values from current year's data, either from strip trials or from performance trials, are useful for making cultivar recommendations for the next year.
Pairwise Comparisons of Cultivars in Strip Trials
One purpose of the trials, especially the strip trials, is to compare the performance between pairs of cultivars. For precise comparison of a particular pair of cultivars, conventional pairwise comparisons can be conducted using data from trials in which both cultivars are tested. Below, we propose a simplistic method for rough comparison of two cultivars based on t-values of BLUP.
Adopting the expression of t-statistics that is used for comparing fixed effects of two cultivars, the difference in BLUP between a pair of cultivars (designated by the subscripts 1 and 2, respectively) can be tested by the following t-statistics:
As an approximation, assume SE1 
SE2, then we have
That is, the t-value for comparing the BLUP of two cultivars can be approximated by the difference between the t-values of the two cultivars. For declaring significant difference between Cultivars 1 and 2 at the 0.05 level, |t12| must be greater than or equal to 2.0 when the degrees of freedom are 60. That is, the following requirement must be met:
Thus, as a rough assessment, two cultivars can be regarded as different in BLUP if their t-values differ by 3.0 or more. For example, based on the 2000 performance trials (Table 2), cultivar RC98 can be regarded as significantly better than cultivar MOR and all cultivars below it. Cultivar MOR, in turn, can be regarded as significantly better than cultivar RON and all cultivars below it. On the basis of the 2000 strip trials (Table 1), Cultivar 2549 can be regarded as significantly better than 2540 and all cultivars below it, whereas 2540 can be regarded as significantly better than cultivar KAR and all cultivars below it.
The Power and Limitation of Strip Trials
Considering that the strip trials are non-replicated, highly unbalanced, and are conducted by farmers under conditions that are less controlled than those under which conventional trials are conducted, it is somewhat surprising that the strip trials produced results comparable, if not superior, to the performance trials (Fig. 1 and 2). However, unlike the performance trial system, which has same evaluation power for all entries, the power of the strip trials is different with cultivars and is strongly dependent on the number of trials in which a cultivar is tested. This can be clearly appreciated by examining the determinants of the t-statistics.
The t-value is simply the ratio of BLUP and its prediction error. Figure 5
demonstrates that the prediction error is a monotone decreasing function of the number of trials in which a cultivar is tested. It declines towards a minimum value as the number of trials increases to infinity. Searle et al. (1992)(Eq. [79], p. 336) provide theoretical explanations for this observation. The prediction error decreases rapidly as the number of trials increases up to about 40, and then levels off. When the number of trials is small, say, 20, an increase in the number of trials is crucial for reducing the prediction error, and hence increasing the evaluation power. Because of the inflation of prediction error, less than 20 trials rarely provided decisive evaluation, except for the extremely good or poor cultivars. Increasing the number of trials beyond 40, however, has little effect on prediction accuracy (Fig. 5).
View larger version (17K):
[in this window]
[in a new window]
|
Fig. 5. Relationship between number of trials and prediction error of the best linear unbiased predictions.
|
|
|
DISCUSSION
|
|---|
This study indicates that strip trials and performance trials, although very different in terms of balance, replication, plot size, management, and implementation, provide similar evaluation of cultivars. This helps answer two questions: whether small-plot performance trials are sufficient to represent conditions in farmers' fields, and whether the highly unbalanced and non-replicated strip trials provide valid data for cultivar evaluation. The two test systems were mutually valid, although in one year the strip trials seemed to be more discriminative than the performance trials, perhaps reflecting the large number of strip trials that cover a wide range of growing conditions. The performance trials, on the other hand, may have the advantage of requiring less seed and being easier to accommodate a large number of cultivars.
Another finding from this study is that the t-statistics of BLUP provide a useful measure of performance reliability of the tested cultivars. The t-statistics and BLUP of the cultivars are always highly correlated for the strip trials (r > 0.93), even though they are highly unbalanced. For the balanced performance trials, they differ only by a constant (i.e., the prediction error SE). However, the t-statistic conveys more information than BLUP because it, per se, is a measure of uncertainty. The greater (or smaller) the t-value of a cultivar, the greater the confidence with which it can be recommended (or discarded). An additional merit of the t-statistics is that it is an intuitive measure of cultivar reliability. Without resorting to any statistical operation, cultivars can be directly divided into three groups: superior (t +2), inferior (t -2), and intermediate or inadequately tested (-2 < t < +2). Furthermore, any pair of cultivars can be roughly compared by their t-values: they can be considered as statistically different if their t-values differ by 3.0. We admit that the grouping of cultivars and the comparison between two cultivars proposed here are only rough measures. We believe, however, that these rough measures are accurate enough for data from yield trials, which involve numerous factors and countless sources of variation, and are appropriate and sufficient for making decisions in cultivar evaluation and recommendation.
The finding that a minimum number of trials is required for decisive evaluation of cultivars is of practical value, considering that many farmers conduct strip trials with the purpose of identifying better cultivars on the basis of their own results and consider results from other farms largely irrelevant. Farmers must be made aware that strip trials are useful for cultivar recommendation only when data are analyzed across trials, and that correct and decisive conclusions may not be possible for cultivars tested in only a few trials. On the other hand, some cultivars, particularly those that are used as checks, were excessively tested in up to 300 trials (Table 1 and Fig. 5). More carefully planned strip trials could make a better use of limited resources.
|
ACKNOWLEDGMENTS
|
|---|
We thank the Ontario winter wheat cooperators, Zorka Szlavnics, Arend Smid, Morley McLean, Mark Etienne, Peter Matthew, and Ellen Sparry for providing the Ontario winter wheat performance trial data, and the Ontario Soil and Crop Improvement Association, the farmers, and seed companies for providing the strip trial data. We also thank Drs. N.C. Stoskopf, D.E. Falk, and three anonymous reviewers for their careful review of the manuscript and valuable suggestions.
Received for publication January 5, 2001.
|
REFERENCES
|
|---|
- DeLacy, I.H., K.E. Basford, M. Cooper, J.K. Bull, and C.G. McLaren. 1996. Analysis of multi-environment trials—a historical perspective. p. 39–124. In M. Cooper and G.L. Hammer (ed.) Plant adaptation and crop improvement. CAB Int., Wallingford, Oxon, UK.
- Eskridge, K.M. 1996. Analysis of multiple environmental trials using the probability of outperforming a check. p. 283–308. In M.S. Kang and H.G. Gauch (ed.) Genotype-by-environment interaction. CRC Press, Boca Raton, FL.
- Littell, R.C., G.A. Nilliken, W.W. Stroup, and R.D. Wolfinger. 1996. SAS system for mixed models. SAS Institute Inc., Cary, NC.
- Piepho, H.P. 1994. Best linear unbiased prediction (BLUP) for regional yield trials: A comparison to additive main effects and multiplicative interaction (AMMI) analysis. Theor. Appl. Genet. 89:647– 654.
- Searle, S.R., G. Casella, and C.E. McCulloch. 1992. Variance components. Wiley, New York.
- Yan, W. 1999. Methodology of cultivar evaluation based on yield trial data—with special reference to winter wheat in Ontario. Ph.D. thesis, Univ. of Guelph, Guelph, ON, Canada.
- Yan, W., L.A. Hunt, Q. Sheng, and Z. Szlavnics. 2000. Cultivar evaluation and mega-environment investigation based on the GGE biplot. Crop Sci. 40:597–605.[Abstract/Free Full Text]
This article has been cited by other articles:
|
|
|
|
 
J. H. Klapwyk and Q. M. Ketterings
Soil Tests for Predicting Corn Response to Nitrogen Fertilizer in New York
Agron. J.,
May 3, 2006;
98(3):
675 - 681.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
W. Yan and I. Rajcan
Prediction of Cultivar Performance Based on Single- versus Multiple-Year Tests in Soybean
Crop Sci.,
March 1, 2003;
43(2):
549 - 555.
[Abstract]
[Full Text]
[PDF]
|
|
|
TOP
ABSTRACT
INTRODUCTION
