Crop Science 40:704-709 (2000)
© 2000 Crop Science Society of America
CROP ECOLOGY, MANAGEMENT & QUALITY
Method for Using Images from a Color Digital Camera to Estimate Flower Number
F.J. Adamsena,
T.A. Coffelta,
John M. Nelsonb,
Edward M. Barnesa and
Robert C. Ricea
a U.S. Water Conservation Laboratory, USDA, ARS, 4331 E. Broadway Rd., Phoenix, AZ 85040 USA
b University of Arizona Maricopa Agricultural Center, Maricopa, AZ USA
fadamsen{at}uswcl.ars.ag.gov
 |
ABSTRACT
|
|---|
In many plants, flowering is conspicuous in the field, but enumerating flowers is labor intensive, especially when flowers need to be counted on a daily basis. Frequent trips into plot areas and the physical contact with the plants can result in mechanical damage to plants, which can affect results. The objectives of this work were to develop methods using color digital images to estimate the numbers of flowers present in a scene captured in a digital image and to do all of the processing in a fully automated mode that would allow the counting of flowers in large numbers of images. Images of lesquerella [Lesquerella fendleri (Gray) Wats.] flowers were made using a color digital camera of field plots during the 1996 to 1997 growing season. An automated system to identify all of the pixels in an image that were flowers and to count the number of flower spots in an image was developed. Processing time for individual images was 3.5 min compared with a minimum of 45 min for manual counts. The automated methods produced results that were highly correlated with the number of flowers in an image as counted by hand. Results of the automated methods accurately tracked the temporal changes in flower number. Multiple counts of the same plants were made by the automated methods without damage to either plants that were counted or the plot. This method has the potential to be used to predict harvest dates from peak flowering, to track the response of flowering to environmental conditions, and to evaluate the effects of cultural practices on flowering.
Abbreviations: B, blue • CCD, charge coupled device • G, green • GIF, graphic interchange file • JPEG, joint photographic experts group • R, red
|
INTRODUCTION
|
|---|
IN MANY PLANTS, flowering is conspicuous in the field. This is true for plants where there is marked difference in color between the flowers and the vegetative growth and/or the flowers are displayed prominently at the top of the canopy as in crops such as canola (Brassica napus L.) and lesquerella. While flowering is an important developmental stage in most crops, enumerating flowers is labor intensive, especially when flowers need to be counted on a daily basis. Under field conditions, frequent trips into plot areas and the physical contact with the plants that is required to make flower counts can result in significant mechanical damage to plants, which can affect results. As a consequence, studies that involve frequent flower counts often are conducted in greenhouse and growth chamber environments (Seddigh and Jolliff, 1994; Williams et al., 1986). These factors combine to limit the size of many studies designed to evaluate factors that affect flowering.
Flowering can be affected by day length, temperature, developmental stage, fertility levels, water stress, cultivar, and other factors (Biarnès-Dumoulin et al., 1996; Coffelt et al., 1989; Duthion et al., 1994; Norberg et al., 1993; Oliva et al., 1994; Seddigh and Jolliff, 1994; Tommey and Evans, 1992; Turner, 1993; Swiader et al., 1994; Sharma et al., 1990). These effects need to be quantified in order to enhance plant growth models, improve management practices, and aid in breeding programs. The development of low-cost color digital cameras that use charge coupled device (CCD) arrays to capture images offer a potential method for measuring flowering in plants with appropriate architecture and flower color.
Dymond and Trotter (1997) used a CCD array to obtain color images of forest and pasture targets from aircraft. They were able to calibrate the system and use it to calculate bidirectional reflectance properties of different targets. Clarke (1997) used a pair of black and white CCD cameras with filters in combination with a thermal imaging system also mounted in an aircraft to estimate water stress in muskmelon (Cucumis melo L.). Both of these systems demonstrated that CCD array systems can be used to determine plant parameters on a large scale.
On a smaller scale, Adamsen et al. (1999) used a color digital camera to measure the greenness of a wheat canopy on a plot basis by calculating the ratio of green (G) to red (R) in each pixel of images representing a 1-m square. Kawashima and Nakatani (1998) used images captured from a color video camera to estimate the chlorophyll content of individual leaves. They used a normalized difference approach applying the blue band (B) as well as the R and G values.
All of the above methods have focused on estimating the amounts and status of the vegetative portion of the plants. The objectives of this work were to develop nondestructive methods using color digital images to estimate the numbers of flowers present in a scene captured in a digital image and to do all of the processing in a fully automated mode that would allow the counting of flowers in large numbers of images.
|
Materials and methods
|
|---|
Images of flowers were made in field plots of an experiment to determine the effects of seeding rate and N fertilizer rate on lesquerella yield at the University of Arizona's Maricopa Agricultural Center, 
40 km south of Phoenix, AZ during the 1996 to 1997 growing season. The soil is a variable Mohall sandy loam (fine-loamy, mixed, superactive, hyperthermic Typic Calciargid) (Nelson et al., 1996). The crop was planted on 15 October. All plots received a preplant application of 45 kg ha-1 of N and 25 kg ha-1 of P as ammonium phosphate-sulfate. Two plots, A and B, were used to develop the flower enumeration system. Plot A was seeded at a rate of 3.8 kg ha-1 with 0 kg N ha-1, and Plot B was seeded at a rate of 7.6 kg ha-1 with 120 kg N ha-1. Additional N fertilizer was applied at flowering on 4 Mar. 1997 as ammonium sulfate. Plots were 3.7 by 15 m in size.
Lesquerella was planted very near the surface and germination was slow. As a result, it was necessary to irrigate the plots four times with minimal irrigations to keep the surface soil moist enough to achieve germination (Table 1)
. Plots were irrigated eight times to replenish soil moisture.
Images were taken 13 times from 9 Apr. 1997 through 22 May 1997 with the same color digital camera (Model D-300L, Olympus America Inc., Melville, NY).1
The camera was positioned 1.6 m above the top of the plant canopy and viewed the same spot in each of 12 plots. The camera was extended 1 m into the plot by an extension arm on the pole which held the camera. This configuration gave a 0.5-m buffer between the edge of the plot and the outside edge of the imaged area. The camera operator never entered the plot area, thereby avoiding damage to the plants. The camera, which had a 1:2.8 5-mm lens, always had a nadir view of the crop. The nominal field of view of the camera was 57 by 42°. The images were acquired between 1030 and 1100 h MST. The camera's built-in flash was used for each image. A white plate with R, G, and B strips was included at the edge of each scene to provide color balance and brightness control.
The camera produced images of 1024 by 768 pixels. Successive quadruplicates in the CCD array recorded 8-bit bands of yellow, magenta, cyan, and green visible light. There was a self-focusing mechanism built into the camera. The data from the four color bands were used via a proprietary algorithm to construct R, G, and B digital values for each pixel in the final image, which was stored in the camera in joint photographic experts group (JPEG) file format. Images were down loaded from the camera to an IBM-compatible personal computer as JPEG files.
The images were converted to GIF (graphic interchange file) format using Paint Shop Pro version 3.12 (JASC Inc., Minnetionka, MN). All of the subsequent processing of the images was accomplished using the image toolbox for MatLab version 4.2c.1 for Windows (Mathworks Inc., Natick, MA). For purposes of calibration, the number of flowers visible in an image was determined by manual counting for Plots A and B, for all dates. Manually determined flower counts from Plots A and B counted for every date were regressed with values determined by the automated method of flower percentage and Euler number using the linear regression function in Quattro Pro for Windows 3.1x version 7 (Corel Corp. Limited, Ottawa, Ontario, Canada). The goodness of fit statistic, root mean squared error (RMSE), or standard error of the Y estimate was also calculated.
|
Results and discussion
|
|---|
The ultimate goal of the image processing we are reporting was to reduce an image to pixels that were flowers and those that were not. The general approach was to keep the processing as simple as possible, which results in what might be viewed as a naive approach to the problem. The preprocessing step of converting an image from JPEG to GIF format resulted in a reduction of the number of colors from potentially millions of colors to a maximum of 256 colors.
The first step in the processing loop was to crop the image from 1024 by 768 pixels to 670 by 670 pixels (Fig. 1, Plate 1a) . Cropping to this size removed the most distorted and off-nadir portion of the image and left an image that represented a 1 by 1 m square, with each pixel representing a 1.5 by 1.5 mm square (12% of the width of an open lesquerella flower). This also reduced the field of view to 36°, which means the outer most pixels were 18° off nadir. The area outlined by the red box in Plate 1a was enlarged in Plate 1b through 1f to allow the reader to see the changes in the image as it was processed. This portion of the image is an area of 100 by 100 pixels or 2.2% of the cropped image. However, the total cropped image of 670 by 670 pixels was used for all calculations.
View larger version (98K):
[in this window]
[in a new window]
|
Plate 1 Results of image processing on the image from Plot A on 15 Apr. 1997. (a) cropped image, (b) area within cropped image outlined in red, (c) outlined area after color depletion, (d) outlined area after color remapping, (e) outlined area after search for yellow spots, (f) outlined area after elimination of non-yellow pixels
|
|
The second step was to change all of the pixels that had G or R values less than 0.85 to black, which left only black and shades of white and yellow in the image (Fig. 1, Plates 1b and 1c). The MatLab software represents the value of each color component in a GIF color matrix in the range of 0 to 1 rather than from 0 to 255 as in most image software. This system has the advantage of being independent of color depth. Color depth is the number of bits used to represent each color in the image. The threshold values of 0.85 for R and G color numbers were determined by detailed examination of images. The thresholds were low enough to include shaded flowers but not so low as to include exposed soil or other plant parts. These values would be expected to differ with various species or possibly with different genotypes within a species. The threshold values also could vary with the type of equipment used to obtain the image.
The third step was to convert the color-depleted GIF image to a RGB image. A RGB image is represented by three arrays of color numbers one for each of the color components of the RGB color model. In contrast, a GIF image is stored as two arrays. The first is an array of references to the second array. The second array is a two-dimensional array containing the color values of RGB components whose maximum size is 256 by 3. The conversion to a RGB image allowed the software in the fourth processing step to remap the colors remaining in the image back to a GIF image that had a maximum of eight primary colors. The colors generated were all combinations of the maximum and minimum values of the RGB color components (Plate 1d). The color matrix from this step was then searched to find the elements that represented yellow, black, and white. These colors were used in subsequent processing steps.
Detailed examination of images after step four showed that many flower pixels are remapped as white pixels. However, when a spot is a flower there is always at least one yellow pixel in the spot. Therefore, the fifth step in the process was to convert any nearest neighbors of a yellow pixel that were white to yellow (Fig. 1). This procedure was carried out twice, first starting in Row 1 Column 1 and proceeding across each row and then to the next row until the entire matrix had been examined. The process was then repeated starting from the last row and column and proceeding in the reverse order of the first pass. Because this is a sequential process, when a yellow pixel is found in a cluster of white pixels, all of the white pixels in the cluster will be converted to yellow before they are tested and all of their white neighbors are changed to yellow as well. Running the process in both directions guarantees that all white pixels in a spot will be converted to yellow if there is at least one yellow pixel in the spot (Plate 1e). In Step 6 (Fig. 1), any pixels that were not changed to yellow during Step 5 were converted to black (Plate 1f). At this point the image was reduced to only two colors; flowers were now all yellow and everything in the original image that was not a flower was black. In order to take advantage of the binary image processing capabilities of the MatLab software, the image was converted to a true binary image in Step 7 (Fig. 1) by constructing a matrix of zeros and ones, where zeros were black pixels and ones were yellow pixels.
The number of flower pixels was determined in Step 8 (Fig. 1) by multiplication of the binary matrix and unit vectors of 1 by 670 and 670 by 1 to give the number of pixels that were flowers. The result was then converted to a percentage of the total number of pixels in the image. The number of flower objects in an image was determined by using the Euler number routine in MatLab with the default settings (Thompson and Shure, 1993). This routine returned the number of objects in a binary image. Euler number values and the flower percentage values were then written to a file, all of the variables cleared, and the routine returned to process the next image. Processing time for the procedure (Fig. 1) was 3.5 min on a Pentium 133 computer with 32 megabytes of memory. Manual counts took at least 45 min and much longer when large numbers of flowers were present. The automated process was set up as a batch job for multiple images, and images from an entire season were processed overnight.
The fate of individual pixels can be followed in Plate 1 from the first processing step to the last as each Plate 1b to 1f represents the same part of the image marked in Plate 1a. There are at least eight spots that were removed in the transition from Plate 1e to 1f. Examination of those areas in Plate 1b show that those areas were either unopened buds or light-colored branch materials. In all cases, none of those areas would have been counted as flowers. The size of flower areas in Plate 1f, the final step, appear to be smaller than the flower areas in Plate 1b, the first step. This indicates that some flower material was not included by the process and some small flowers or flowers in the wrong orientation were not counted.
Results of flower counts on Plot A and the percentage of flower pixels in the image show very similar temporal trends (Fig. 2) . The difference in patterns at the early sampling dates could be the result of larger flowers during early flowering or better exposure of the flowers in the images (i.e., fewer flowers partially covered by other flowers or plant parts). The peaks in flowering in mid April and early May followed irrigations on 18 Apr. 1997 and 5 May 1997.
View larger version (25K):
[in this window]
[in a new window]
|
Fig. 2 Temporal changes in flower pixels, Euler number, and flower number of lesquerella in Plot A in 1997
|
|
The Euler numbers and flower numbers for Plot A also showed similar temporal trends (Fig. 2). In general, Euler numbers were greater than flower numbers. This may have occurred because in some cases individual flowers show up in the image as more than one spot. Also, it may be an artifact of the Euler number procedure. There was better agreement in slope of early flowering between the two counting methods than between flower number and percentage of flower pixels. Euler number did not show as great an increase in value following the 5 May 1997 irrigation as did flower number and percentage of flower pixels.
Regression analyses on the percentage of flower pixels vs. flower number and Euler number vs. flower number yielded similar results (Fig. 3 and 4)
. Coefficients of determination (r2) for the regressions were greater than 0.8 for both automated flower estimates vs. flower number. Linear regressions for both automated methods tended to estimate higher values at low flower number, and there was more divergence in flower number with either of the automated methods at high flower number. However, the estimates are precise enough to track temporal trends accurately (Fig. 2).
All of the counting described was carried out without any direct contact with the plants. Because of the plot layout images could be obtained without entering the plot area so traffic damage in the plots was avoided as well. This resulted in multiple flower number estimates on the same plants without any damage to either the plants that were counted or to the plots.
The authors have applied this method to flowering of canola, both B. rapus and B. napus L., crambe (Crambe L.), and vernonia (Vernonia galamensis [Cass.] Less.). In the case of canola, the method can be used without change. Vernonia has a bluish purple flower and crambe a white flower, which required different threshold values. The canopy architecture of crambe is similar to that of alfalfa (Medicago sativa L.).
Potential uses of flowering data include tracking peaks in flowering after irrigation or rain events, which can be an important plant growth model input; predicting harvest dates based on peak flower times; documenting effects of fertility and other cultural practices; identification of day-neutral and other flowering characteristics of various crosses in breeding programs; determining flower counts for use in measuring reproductive efficiency; and establishing the effects of stress on reproduction. The automated methods described above have the potential to make possible intensive sampling of flowering in a variety of crop plants.
|
NOTES
|
|---|
1 Trade names are included for the benefit of the reader and imply no endorsement or preferential treatment of the products listed by the USDA. 
Received for publication December 22, 1998.
|
REFERENCES
|
|---|
- Adamsen F.J., Pinter P.J., Jr, Barnes E.M., LaMorte R.L., Wall G.W., Leavitt S.W., Kimball B.A. Measuring wheat senescence with a digital camera. Crop Sci. 1999;39:719-724.[Abstract/Free Full Text]
- Biarnès-Dumoulin V., Denis J.-B., Lejeune-Hénaut I., Etévé G. Interpreting yield instability in pea using genotypic and environmental covariates. Crop Sci. 1996;36:115-120.[Abstract/Free Full Text]
- Clarke T.R. An empirical approach for detecting crop water stress using multispectral airborne sensors. HortTech 1997;7(1):9-16.
- Coffelt T.A., Seaton M.L., VanScoyoc S.W. Reproductive efficiency of 14 virginia-type peanut cultivars. Crop Sci. 1989;29:1217-1220.[Abstract/Free Full Text]
- Duthion C., Ney B., Munier-Jolain N.M. Development and growth of white lupin: implications for crop management. Agron. J. 1994;86:1039-1045.[Abstract/Free Full Text]
- Dymond J.R., Trotter C.M. Directional reflectance of vegetation measured by a calibrated digital camera. Appl. Optics 1997;36:4314-4319.
- Kawashima S., Nakatani M. An algorithm for estimating chlorophyll content in leaves using a video camera. Ann. Bot. (London) 1998;81:49-54.[Abstract/Free Full Text]
- Nelson J.M., Dierig D.A., Nakayama F.S. Planting date and nitrogen fertilization effects on lesquerella production. Ind. Crops Products 1996;5:217-222.
- Norberg O.S., Seddigh M., Jolliff G.D., Feiz T.E. Flower production and honey bee density effects on meadowfoam. Crop Sci. 1993;33:108-112.[Abstract/Free Full Text]
- Oliva R.N., Steiner J.J., Young W.C.I. White clover seed production: II. Soil and plant water status on yield and yield components. Crop Sci. 1994;34:768-774.[Abstract/Free Full Text]
- Seddigh M., Jolliff G.D. Light intensity effects on meadowfoam growth and flowering. Crop Sci. 1994;34:497-503.[Abstract/Free Full Text]
- Sharma K.P., Dybing C.D., Lay C. Soybean flower abortion: Genetics and impact of selection on seed yield. Crop Sci. 1990;30:1017-1022.[Abstract/Free Full Text]
- Swiader J.M., Sipp S.K., Brown R.E. Pumpkin growth, flowering, and fruit response to nitrogen and potassium sprinkler fertigation in sandy soil. J. Amer. Soc. Hort. Sci. 1994;119:414-419.
- Thompson C.M., Shure L. Image processing toolbox for use with MatLab. Natick, MA: The MathWorks, Inc., 1993.
- Tommey A.M., Evans E.J. The influence of pre-floral growth and development on the pathway of floral development, dry matter distribution and seed yield in oilseed rape (Brassica napus L.). Ann Appl. Biol. 1992;121:687-696.
- Turner L.B. The effect of water stress on floral characters, pollination and seed set in white clover (Trifolium repens L.). J. Exp. Bot. 1993;44:1155-1160.[Abstract/Free Full Text]
- Williams I.H., Martin A.P., White R.P. The pollination requirements of oil-seed rape (Brassica napus L.). J. Agric. Sci. (Cambridge) 1986;106:27-30.
TOP
ABSTRACT
INTRODUCTION
