Evaluation of the New ASTER Global Digital Elevation Model James A. Slater1, Barry Heady2, George Kroenung2, William Curtis3, Jeffrey Haase3, Daryl Hoegemann3, Casey Shockley4, Kevin Tracy3 National Geospatial-Intelligence Agency May 15, 2009 Revised July 29, 2009 1InnoVision Directorate, Basic and Applied Research Office, Reston, Virginia 2Source Directorate, Geoint Integration Office, St. Louis, Missouri 3Source Directorate, Geoint Sciences Office, St. Louis, Missouri 4Analysis & Production Directorate, Functional Operations Group, St. Louis, Missouri INTRODUCTION Japan’s Ministry of Economy, Trade and Industry (METI) in collaboration with the U.S. National Aeronautics and Space Administration (NASA) have sponsored the development of a new global digital topographic model derived from multi-spectral imaging data collected with the Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) onboard the Terra satellite. [NASA Jet Propulsion Laboratory, 2009] Almost 10 years of data acquired from 1999-2008 have been reprocessed with new software by Sensor Information Laboratory Corp. (SILC) under the direction of Dr. Hiroyuki Fujisada in Japan. A preliminary version of the new global digital elevation model (GDEM) was released to a number of organizations worldwide for evaluation. The evaluation effort is being coordinated by the U.S. Geological Survey (USGS) EROS Data Center. Due to its extensive global holdings of elevation data and its experience producing Digital Terrain Elevation Data (DTED®), and in particular, the Shuttle Radar Topography Mission (SRTM) data, the National Geospatial-Intelligence Agency (NGA) was invited to provide a global analysis of the new GDEM. NGA chose 20 sites in 16 countries for this evaluation (See Figure 1). Except for one site in Alaska, none of these sites is in the United States, since the USGS planned to devote its resources to an analysis of the new GDEM over the U.S. This allowed NGA to focus its attention on other continents. NGA utilized two types of data for comparison with the ASTER GDEM. The first is DTED®, which is maintained at two levels of product resolution, DTED® 1 (3 arc second) and DTED® 2 (1 arc second). In total, NGA DTED® 1 and 2 holdings cover greater than 90% of the Earth’s surface. The DTED® 2 are almost completely comprised of SRTM source data that were collected during the 2000 STS-99 mission. [Farr et al., 2007; Slater et al., 2006] Above 60° N latitude, where no SRTM data are available, the DTED® used in the evaluation were derived from electro-optical imagery and cartographic sources. Additional sources of DTED® were utilized at two of the sites and are documented below. The second type of data used in the analysis is “control point” data. These are discrete points photogrammetrically derived from satellite imagery. Detailed descriptions of the NGA reference data are given later in this report. The evaluation performed by NGA consists of standard DEM-to-DEM comparisons and DEM-to-control-point comparisons as well as detailed visual analyses of the data. The visual analysis is essential for discovering artifacts in the data that are usually hidden in the aggregate statistics derived from the almost 13 million elevation values contained in a 1° x 1° cell at 1 arcsec resolution. The following sections of this report describe the ASTER GDEM data that NGA received, the NGA reference data, the evaluation objectives and methodology, the results of the statistical analysis and the visual analysis, conclusions and recommendations. ASTER DEM DATA The Level 1-A ASTER data were compiled and processed by SILC using new software designed for this project. All the existing historical data from 1999-2008 were screened for cloud problems, correlation problems, and outliers based on comparison with a reference DEM or the average of the stacked scenes for each pixel. The data used to produce the GDEM were from VNIR Band 3N (0.76-0.86 micron wavelengths). ASTER’s geographic area coverage extends Figure 1. NGA evaluation sites (SRTM source = green). from 83° S to 83° N latitudes. The data were provided in 1° x 1° tiles at a one arcsec (30 m) resolution, such that each tile contains 3,601 x 3,601 pixels. The ASTER GDEM is referenced to the World Geodetic System 1984 (WGS 84) and the elevations are computed with respect to the WGS 84 EGM96 geoid. The elevation value at each pixel is the unweighted average of the accepted scenes in the stack. Ocean and sea elevations are set to 0 based on low observed radiance (DN) values and a GTOPO30 sea water database. It appears that no specific editing was done to identify and delineate lakes and rivers. A “Number” file was also provided with each tile, which gives the actual number of scenes that were averaged to generate the elevation value at each pixel. The Number file also indicates if a pixel was corrected using a reference DEM or a pixel contains an interpolated value. The accuracy objective at the 95% confidence level for the new ASTER DEM was given as 20 m vertical error and 30 m horizontal error. (At a 90% confidence level, these equate to 16.8 m vertical error and 26.3 m horizontal error.) NGA REFERENCE DATA NGA selected 20 geographic areas as a representative worldwide sample of topography for which reference elevation data existed to compare with the ASTER DEM. These areas are comprised of 284 1° x 1° cells with varied terrain relief and elevations ranging from sea level to over 6,700 m (See Table 1). Two types of reference data were used for this evaluation: (1) DTED® for DEM-to-DEM comparisons and (2) ground control points (GCPs) for point-to-point comparisons within each cell. DTED® and GCP coordinates are referenced to WGS 84 and the Table 1. NGA Reference DEM Data Characteristics (a) Source: SRTM DTED® 2. (b) Photogrammetrically-derived DTED®; Cartographically-derived DTED®; Interferometric Synthetic Aperture Radar (IFSAR) DEM. (c) L.E. is linear error; C.E. is circular error. elevations are defined as heights above the WGS 84 EGM96 geoid. Each of the specific reference data sources is described below. Digital Terrain Elevation Data (DTED®) Most of the data for the DEM comparisons are SRTM DTED® 2; however, other sources of photogrammetrically-derived DTED® 2 were the only data used in three areas – Russia sites 1 and 2 and Canada, and cartographic source DTED® 2 were the sole source of reference data used in two other areas – Russia site 3 and Alaska. Multiple sources of DTED® were available in a few areas. Afghanistan site 2 and Korea had DTED® from SRTM and photogrammetrically-derived sources, and Afghanistan site 1 had a commercial source of IFSAR available in addition to the SRTM data. The DTED® DEM products represent standard data products held in NGA’s data warehouse. All of the DTED® data have the same format and basic data content, although some differences may exist in the characteristics of DTED® derived from different sources. In general, water bodies have been post-processed such that ocean elevations are set to zero, lakes are flattened to a constant value, and double-line drains (rivers) are stepped down to fit the terrain, in accordance with the DTED® standard. [NIMA, 2000] Accuracy is derived on a cell by cell basis and is driven by the source accuracy. For latitudes of 50° or less, the DTED® are gridded at 1 arcsec x 1 arcsec spacing resulting in a 3,601 x 3,601 array of points; between 50 and 70 degrees, DTED® has 1 x 2 arcsec spacing (3,600 x 1,801 array of points); For the latitude intervals from 70-75 degrees, 75-80 degrees and 80-90 degrees, DTED® has 1x3 arcsec spacing, 1x4 arcsec spacing, and 1x6 arcsec spacing, respectively. SRTM DTED® 2 were derived from data acquired with a C-band interferometric synthetic aperture radar (IFSAR) system flown on the Space Shuttle and represent the radar reflective surface, which is typically similar to optical reflective surface data, except that the radar may penetrate the tree cover to some extent, depending on a number of other factors. NGA has a version of these data where the original voids have been filled as much as possible using other existing DEM sources and some interpolation in uniformly flat areas. The data have also undergone additional editing to remove spikes and wells exceeding a 60-meter threshold and anomalous elevation areas caused by phase unwrapping errors. This data set was chosen as the primary comparison source for the ASTER GDEM evaluation where available. Vertical accuracy for SRTM DTED® is reported for each cell, but horizontal accuracy is available only on a continental basis. Photogrammetrically-derived Stereo Imagery Data Photogrammetrically-derived DTED® 2 are derived from electro-optical stereo imagery. These data are typically edited to a bare earth surface, except in those areas where the tree cover is too dense to see the underlying surface; water bodies are flattened, and spikes and wells are removed. Since the surface has typically been edited to a bare earth, the spike/well threshold is typically in the 10-15 meter range. Horizontal and vertical accuracies for photogrammetrically-derived DTED® are estimated and reported independently for each cell. SPOT 5 High Resolution Stereoscopic (HRS) Data SPOT 5 HRS DTED® 2 are derived from SPOT stereo optical imagery. These data have been edited very similarly to the photogrammetrically-derived DTED® 2 noted above. Horizontal and vertical accuracies for SPOT 5 DTED® are estimated and reported independently for each cell. The estimated absolute vertical accuracy is 2-5 m and horizontal accuracy is 14-16 m (90% values). These data were used for GCP comparison purposes only in the Libya sample area. Cartographic Source Cartographic source DTED® 2 was used in areas above 60° N when other sources were not available. The Alaskan DTED® 2 data were derived from USGS 1:64,000 scale maps and are of reasonable quality and possess the same water finishing characteristics as the other DTED® data sets. The Alaska data are essentially the same as the USGS National Elevation Dataset (NED). These data are recognized to have horizontal errors larger than those cited in the DTED® specification. The Russia site 3 data were compiled from 1:50,000 scale native map source and finished to standard DTED® product specifications, including water body processing. Commercial IFSAR Data A commercially-produced IFSAR source DEM was used for comparison in one area (Afghanistan site 1). These data were produced as a mission specific data set for NGA at a 5 m spatial resolution and represent the radar reflective surface elevations. Ground Control Points (GCPs) GCPs are derived photogrammetrically during the triangulation processing of the stereo pair optical source and represent discrete 3-dimensional reference points extracted from the stereo imagery. There are typically approximately 200 of these points per 1° x 1° area, although that number can vary based on the terrain type and geographical location. These points typically do not directly horizontally coincide with the location of a DEM post, so the gridded DEM points surrounding the control point are bilinearly interpolated to estimate the elevation value of the DEM at the control point location. The accuracy of these photogrammetrically-generated control points is typically better than 10 m vertically and 10 m horizontally at a 90% confidence level. EVALUATION METHODOLOGY Data evaluation was accomplished using standard DEM analysis techniques to generate statistical and visual analyses of the data. Statistical analysis was based on direct DEM-to-DEM comparison and DEM-to-control point comparisons. Standard DEM analysis tools were used to quantify the nature of DEM data (spike/well, artifact analysis, etc.). NGA masked out any water, alternate source fill and interpolated regions from the comparisons to avoid undue influence from these areas in the data comparisons and resulting statistics. NGA also resampled the DTED® data to match the ASTER Global DEM resolution. This was only necessary above 60° N where the DTED® data are gridded at a 1 by 2 spacing (3,601x1,801) versus the ASTER Global DEM 3,601x3,601 resolution. NGA also performed visual analysis of the data to glean additional information by creating shaded relief and elevation based color map displays. Statistics can sometimes be misleading when evaluating a DEM in that serious data problems can be averaged out in the statistics when comparing almost 13 million individual data points per one-degree cell. Visual analysis is usually required to detect data artifacts that may be present which could sometimes have implications for DEM applications. Extreme random vertical noise, mosaic artifacts, excessive spikes and wells and unreliable terrain depiction are examples of these types of phenomenon. EVALUATION RESULTS ASTER GDEM Comparison with Reference DEMS Table 2 shows the results of comparing the new ASTER GDEM to NGA reference DEMs for each of the 20 geographic areas selected. These areas vary in size from 1 or 2 cells in the case of two Russian sites and the Philippines to 25 cells for seven of the sites. Each of these areas represents a set of contiguous cells and the statistics for each site represent the whole contiguous area. The first thing to note is that, in almost every area, the mean ASTER elevations are lower than the reference DEM elevations. It is known that the SRTM DTED® are generally biased high in areas with foliage due to the reflective surface nature of the data, but as far as we know, the ASTER elevations were not adjusted to bare earth and thus should have a similar bias. Therefore the lower mean ASTER elevations relative to SRTM cannot be accounted for by differences in the measured surface of the Earth. The 90% confidence intervals about the mean (ASTER-SRTM) DEM differences are completely negative in Bolivia (-19.0, -2.8) and Nigeria (-18.0, -8.6) or in most cases range from negative to positive as in Bosnia (-13.1, 5.1). The results for the non-SRTM reference DEMs vary by area. The Afghanistan site 1 statistics computed using the IFSAR DEM have a mean difference and 90% error comparable to the SRTM statistics; the absolute value of the (ASTER-IFSAR) mean elevation difference is the same as in the SRTM case, except now the ASTER GDEM is above the IFSAR DEM and the linear error is smaller. The latter could be a function of a lower noise level in the IFSAR DEM than the SRTM DEM although that is not apparent from the estimated vertical errors (7 m) given for this IFSAR DEM in Table 1. Similar results are seen using the photogrammetrically-derived DTED® in Afghanistan site 2 where the mean elevation difference is almost 1 m less than the SRTM case and the linear error is smaller. Korea is the only other area where statistical comparisons were made for SRTM and photogrammetrically-derived DTED®. Here there is no mean difference between the ASTER and non-SRTM DTED®, but the ASTER elevations are on the average 6 meters lower than SRTM data. The vertical error in the photogrammetrically-derived case (14.9 m) is significantly larger than the error in the SRTM case (10.8 m). For the Russia sites 1 and 2, only photogrammetrically-derived DEMs were available. The mean DEM difference for site 1 is comparable to other sites (ASTER 4.1 m lower than reference source with a 90% error of 12.4 m). The minimum and maximum elevations in the three ASTER cells evaluated at these sites differ dramatically from the values in the reference DTED®. For example, in site 1, the maximum DTED® elevation is 990 m compared to the maximum ASTER elevation of 2,131 m, and similarly in site 2, the maximum ASTER elevation is 2,944 m compared to 706 m for the reference source. (ASTER – DTED®) DEM differences range from a minimum of -771 m to a maximum of 2,590 m. The results for the one cell that constitutes site 2 are much worse than those for site 1 for both the mean offset (-11.7 m) and its variability (29.4 m) with respect to the reference data. The DEM evaluations of Russia site 3 and Alaska were based on cartographic source DTED®. The reference data are not very good; Russia site 3 DTED® have vertical and horizontal uncertainties of 17 m and 27 m, respectively, so the comparison statistics (-10 m mean elevation bias and 16 m vertical error for the DEM difference) reflect this inaccuracy in addition to the error in the ASTER GDEM. DEM comparison statistics for Russia site 3 are roughly comparable to the other Russian sites. The results for Alaska, although not very good relatively speaking, are also partly attributable to the 16-33 m vertical error estimated for the cartographic reference DTED®. It is worth noting that the original pre-release data that NGA evaluated had huge anomalous areas in Russia sites 2 and 3 that resulted in vertical errors of 160-550 m over these areas. Much of this data has been voided out in the public release version of the ASTER GDEM, thus improving the aggregated statistics; however, these sites still contain extraneous elevation data as indicated by the kilometer-magnitude of the minimum and maximum differences with the reference DEMs cited above. Table 2. DEM Differences for NGA Evaluation Areas Geographic Area No. of Cells Mean Elevation Difference ASTER DEM – Reference DEM SRTM Other Source Mean (m) 90% L.E. (m) Mean (m) 90% L.E. (m) Afghanistan-1 8 -0.6 9.8 0.6 (a) 7.2 Afghanistan-2 9 -1.7 5.4 -0.9 (b) 3.7 Argentina 25 -5.2 6.5 Australia 25 -4.7 7.9 Bolivia 4 -10.9 8.1 Bosnia 15 -4.0 9.1 Canada 6 -14.0 (b) 16.2 China-1 25 -7.9 15.2 China-2 25 -4.1 11.1 Iraq 16 -5.0 6.3 Kazakhstan 25 -5.5 6.8 Korea 12 -6.0 10.8 0.0 (b) 14.9 Libya 4 0.0 11.0 Nigeria 25 -13.3 4.7 Philippines 2 -6.6 14.6 Russia-1 2 -4.2 (b) 12.4 Russia-2 1 -11.7 (b) 29.4 Russia-3 20 -9.7 (c) 15.7 Thailand 10 -8.6 8.8 U.S.A.-Alaska 25 -10.0 (c) 25.1 (a) Commercially-produced IFSAR DEM. (b) Photogrammetrically-derived DTED® 2. (c) Cartographic source DTED® 2; Alaska based on USGS NED. Ground Control Point Analysis The ASTER GDEM and the reference DEMs were independently compared to the GCPs available in each geographic area. As noted earlier, the gridded DEMs were interpolated to estimate the DEM elevation value that coincided with the GCP’s horizontal coordinates. A summary of the mean elevation differences of the DEMs with the GCPs and the 90% vertical error estimates is given in Table 3. In every area, the mean ASTER elevations are lower than the control points, ranging from close to zero difference (Australia and Libya) to 11-16 m differences (Canada, Nigeria, Russia). The Canadian (34 m) and Russian sites (34, 27, and 25 m, respectively) also exhibit the largest variability. These results can be compared to the SRTM-GCP differences in 15 of the areas. In the SRTM case, most of the mean differences tend to be positive, i.e. placing the SRTM elevations above the GCP elevations. The magnitude of the ASTER variability is two to three times that of the SRTM variability with respect to the control points. The exceptions are Kazakhstan, where the mean SRTM elevations are lower than the GCPs by about 3 m, and Bolivia, where the ASTER elevations are more accurate and less noisy than the SRTM DTED® with respect to the GCPs. In the Bolivia case, photogrammetrically-derived elevations are significantly better than the ASTER GDEM with respect to the ground control, and confirm the poorer quality of the SRTM data in this area. This may be a function of Table 3. Comparison of ASTER and Reference DEMS with Ground Control Points for NGA Evaluation Areas (a) IFSAR DEM. (b) Photogrammetrically-derived DTED® 2. (c) SPOT 5 DTED® 2. (d) Cartographic source DTED® 2. (e) 90% error taken directly from frequency distribution plot of the absolute value of (DEM-GCP) in cases where the distribution did not appear to be approximately normal. In all other cases, 90% figures are computed by multiplying the calculated standard deviation of the differences by 1.6449. See the Appendix for all the frequency distribution plots. the relatively small number of control points in Bolivia (79) and the fact that 67 of them are in just one of the four cells but it is indeterminate at this point. In six areas (Afghanistan sites 1 and 2, Bosnia, Iraq, Korea, and Libya), NGA had DTED® from sources other than SRTM, allowing a second independent comparison with the GCPs. The IFSAR DEM source in Afghanistan site 1 and the DTED® source in Afghanistan site 2 are both offset on the average one meter or less from the GCPs and have a smaller vertical error than SRTM and ASTER with respect to the GCPs. Similar results can be seen for Iraq, Korea and Libya using the photogrammetrically-derive DTED® and SPOT 5 data sources. In Bosnia, the reference DTED® 2 is much closer to the GCPs than the ASTER GDEM (half the vertical error and smaller mean offset), but has a larger vertical error than the SRTM DTED®. Five areas at latitudes above 60° N – the three Russia sites, Canada and Alaska – have no SRTM DTED®. Cartographic source was available for Russia site 3 and all the other areas had photogrammetrically-derived DTED®. Except for Alaska, the results confirm the observations made for the areas with SRTM DTED®. The mean differences and vertical errors with respect to the ground control points are much smaller for the reference DEMs than for the ASTER GDEM. The mean difference of the Alaska DTED® and the GCPs is much smaller than the ASTER-GCP difference, but the error is the same (24 m). Figures 2a through 2p are a graphic illustration of the typical variability of the ASTER GDEM compared to the reference DEM. The eight representative geographic areas selected are Alaska, Argentina, Bolivia, Bosnia, China sites 1 and 2, Kazakhstan, and Nigeria. Each plot shows the (DEM-GCP) difference as a function of elevation for all the GCPs in the area. The plots also reveal the distribution of the control points over the elevation range of the data in each area, which can be discerned from the density of the lines representing each control point. The key observations that can be made from these plots are: With respect to the GCPs, there is a clear negative bias in the ASTER elevations and in most cases a positive bias in the reference DEMs (SRTM, Photogrammetrically-derived, NED). There is no obvious relationship between the DEM-GCP differences and actual elevation for either ASTER or the reference DEMs. The variability (noise) of the ASTER data is much greater than that of the corresponding reference DEMs, except for Alaska and Bolivia. The reference DEMs for Alaska and Bolivia appear to be of poorer quality than the other reference DEMs as reflected in the GCP comparisons. The vertical errors of these DEMs are equal to or greater than those of the corresponding ASTER GDEM, and the Bolivian SRTM DEM seems to have a significant bias. Figure 2a. ASTER-GCP elevation differences vs. DTED elevation – Alaska. Figure 2c. ASTER-GCP elevation differences vs. SRTM elevation – Argentina. Figure 2e. ASTER-GCP elevation differences vs. SRTM elevation – Bolivia. Figure 2b. DTED-GCP elevation differences vs. DTED elevation – Alaska. Figure 2d. SRTM-GCP elevation differences vs. SRTM elevation – Argentina. Figure 2f. SRTM-GCP elevation differences vs. SRTM elevation – Bolivia Figure 2g. ASTER-GCP elevation differences vs. SRTM elevation – Bosnia. Figure 2i. ASTER-GCP elevation differences vs. SRTM elevation – China site 1. Figure 2k. ASTER-GCP elevation differences vs. SRTM elevation – China site 2. Figure 2h. Photo. DTED-GCP elevation differences vs. DTED elevation – Bosnia. Figure 2j. SRTM-GCP elevation differences vs. SRTM elevation – China site 1. Figure 2l. SRTM-GCP elevation differences vs. SRTM elevation – China site 2. Figure 2m. ASTER-GCP elevation differences vs. SRTM elevation – Kazakhstan. Figure 2o. ASTER-GCP elevation differences vs. SRTM elevation – Nigeria. Figure 2n. SRTM-GCP elevation differences vs. SRTM elevation – Kazakhstan. Figure 2p. SRTM-GCP elevation differences vs. SRTM elevation – Nigeria. Number of ASTER Scenes Used in GDEM A file was provided with the GDEM that gives the number of stacked ASTER scenes used to compute the elevation value in each pixel. The number of scenes varies from one for some pixels to over 30 in other cases. Figures 3a through 3d show the frequency distribution of the number of passes (scenes) that were averaged per pixel for typical cells in the areas that NGA evaluated. Subsequent to NGA’s analysis of the pre-release version of the GDEM, the Japanese team modified the Russian cells by voiding out some of the anomalous elevation data; however, they did not adjust the Numbers file containing the number of stacked scenes used in the GDEM. The data for three cells in Russia Site 3 in Figure 3d are based on the number of scenes per pixel in the unvoided data and don’t correspond to the public released version of the GDEM, but they do provide useful insight into the relative quality of the Russia elevations compared to other areas that NGA sampled. There is a significant difference between cell N63E043 in Russia Site 3 in Figure 3d compared to Afghanistan Site 2 cells N29E064 and N31E065 and the two Bolivia cells in Figure 3a. In the Russia cell, 86% of the elevation values are based on only 1 or 2 scenes (see Figure 4), while in the Afghanistan and Bolivia cells, 99% of the elevation values are based on 10 or more scenes. It can also be observed that there is a noticeable difference in coverage between cells in the same geographic area. See for example Afghanistan Site 2 cell N29E066 versus the other two Afghanistan cells in Figure 3a. Also, compare the two adjacent Korea cells in Figure 3c. There are also regional disparities in the number of passes used. Australia and Russia cells shown in Figures 3b and 3d, respectively, have relatively low numbers of passes per pixel. These illustrations are intended to show the pixel-by-pixel and cell-by-cell variations in the DEM data. Examination of the pass statistics for all 20 cells in Russia Site 3, for example, might allow a connection to be made between the low number of passes and the poor DEM accuracy (Tables 2 and 3), but that was not done for the present assessment. Accuracy Assessment of the ASTER GDEM Based on the GCPs Noting that (1) the GCPs are independent of the ASTER data, (2) the vertical error of the GCPs is estimated to be no greater than 10 m (90% confidence), and (3) the vertical error (90% confidence) of the ASTER-GCP elevation differences has already been computed in Table 3, an estimate of the ASTER vertical error can be obtained using the following statistical relationship: Variance (ASTER-GCP) = Variance (ASTER) + Variance (GCP) or Variance (ASTER) = Variance (ASTER-GCP) – Variance (GCP). Using the above relationship, the estimated ASTER vertical errors were computed and are shown in Table 4. Since the same maximum GCP error is assumed for all areas, it accounts for a constant portion of the ASTER-GCP difference error in all areas. In reality, the GCP errors are considerably better than 10 m in some areas, which would effectively increase the estimated ASTER error in those areas. The estimated ASTER error in the Bolivia area (0.0) is obviously incorrect. Note that Bolivia was the only area where the SRTM-GCP difference error (15.0 m) was greater than the ASTER-GCP difference error (10.0 m) in Table 3. It may be indicative of a poor set of ground control points rather than poor SRTM data. Assuming that this is perhaps a best case scenario for an estimate of the ASTER vertical error, the values in Table 4 can be compared to the ASTER GDEM specification of 16.8 m vertical error at 90% confidence (equivalent to a 95% value of 20 m). Twelve of the sample areas meet the specification while the other eight do not. Figure 3a. Distribution of the number of passes (scenes) used per pixel in the ASTER GDEM for selected cells in Afghanistan and Bolivia. Figure 3c. Distribution of the number of passes (scenes) used per pixel in the ASTER GDEM for selected cells in China Site 1, Kazakhstan and Korea. Figure 3b. Distribution of the number of passes (scenes) used per pixel in the ASTER GDEM for selected cells in Australia and Nigeria. Figure 3d. Distribution of the number of passes (scenes) used per pixel in the ASTER GDEM for selected cells in Russia Site 3. (Based on pre-release version of GDEM) Figure 4. The number of scenes (passes) used to compute ASTER GDEM elevation values in Russia Site 3 from the ASTER “Numbers” file. The red, green and blue areas represent 1, 2 and 3 scenes, respectively. The table at the right shows the number of stacked scenes (“Value”) used to compute the elevation pixels, the color code in the graphic, and the actual number of pixels corresponding to a given number of scenes. At the location of the faint crosshair in the red area at the lower right of the graphic, the ASTER GDEM elevation is 3,883 m higher than the NGA reference DTED®. Large parts of the red areas in this figure were voided out in the GDEM version released to the public. Topographic Artifacts and Anomalies in the ASTER GDEM After reviewing the ASTER DEM in the 20 selected geographic areas, NGA observed some common and recurring artifacts and anomalies in all of these areas. Figures 5 through 16 illustrate many of these artifacts in the form of shaded relief images of the ASTER GDEM alongside the reference DEM. The general problems can be grouped into the following categories: Poor resolution Noisy data Mosaicking artifacts Poorly fit void-fill data Poor coastline definition and water body identification Landform artifacts Pits and spikes It was expected that the 30-meter post spacing of the ASTER GDEM would yield visible resolutions similar to those of the SRTM DTED® 2 and SPOT 5 HRS DEMs that have the same pixel size. This is apparently not the case. The ASTER GDEM has a lower resolution than the comparable reference data as can be seen in Figures 5, 10 and 15. In addition, the ASTER GDEM is considerably noisier than the reference DEMs (see Figure 6). Table 4. Estimated ASTER GDEM Error Based on GCPs (a) 90% error taken directly from frequency distribution plot of the absolute value of (DEM-GCP) in cases where the distribution did not appear to be approximately normal. In all other cases, 90% figures are computed by multiplying the calculated standard deviation of the differences by 1.6449. Since each pixel is the average of the useable stacked scenes over that pixel location and varies from 1 to over 30 in the data that NGA examined, it is a challenge to merge these pixels seamlessly into a terrain model. It was observed that many areas in the GDEM have small ridge-like structures forming closed loops or open linear features across the terrain (See Figure 8). By comparing the locations of these features with a graphical display of the number of scenes used in a given geographic area, it was discovered that the ridge-like features correspond to the boundaries between areas with different numbers of scenes used to produce the GDEM. They are in effect mosaicking artifacts and vary from 1 to 12 m in height. NGA also noted that the edges of adjacent one-degree cells do not match seamlessly, although the specifications supposedly included cell boundary matching. There are numerous instances where void areas in the ASTER GDEM were filled with alternate data sources as in Figure 9, leaving obvious discontinuities at the boundary of the ASTER data and the fill data. Unlike DTED® products, the data processing plan for the ASTER GDEM did not include any detailed water body delineation and elevation adjustment. As a result, inland water bodies are generally undefined or poorly displayed in the topography (see Figure 10). Shorelines were also misrepresented and false islands were created in the areas that NGA evaluated (see Figure 11). Figures 12, 13 and 14 show several landform features that appear in the ASTER GDEM but do not exist in the reference topography. These include trenches, raised linear features that may correspond to roads, and major land structures over an extended area for which there is no evidence in the reference DEM. Lastly, the ASTER GDEM has many pits and spikes in the data that have not been filtered out. Figure 15 shows a number of pits that occurred along ridges in one area and Figure 16 shows an area with a large number of raised surface features containing pits. Neither the pits nor the raised features appear in the reference SRTM DTED®. In addition to these examples, 11 of the 25 cells in Alaska and numerous cells in Kazakhstan have very large spikes and wells that remain in the ASTER GDEM. Figure 5. Shaded relief of major topographic features comparing ASTER GDEM (left) and SRTM DTED® (right) for China cell N34 E085. The ASTER GDEM is noisier and contains numerous pits. Figure 6. Shaded relief showing high noise in ASTER GDEM (left) compared to SRTM DTED® (right) for a sample Bolivia cell (at S19 57 27 W006 48 00). Figure 7. Shaded relief showing resolution differences and apparent mosaic seam for ASTER GDEM (linear feature, center left) compared to the reference SRTM DTED® (right) for Iraq cell N33 E42. Figure 8. Shaded relief showing ridge-like artifacts in ASTER GDEM (right) compared to SRTM DTED® (left), apparently resulting from the mosaicking process for sample cell in Korea (at N37 54 56 E125 56 18). Figure 9. Shaded relief illustrating poorly fit fill data in the ASTER GDEM (left) compared to NGA DTED® (right) as well as small ridge-like mosaicking artifacts in the ASTER GDEM for a cell in Alaska (at N64 50 34 W147 28 30). Figure 10. Shaded relief of a Thailand water body in the SRTM DTED® (left) and the same area in the ASTER GDEM (right) (at N18 56 14 E099 07 03). Note also the apparent better resolution of the SRTM data relative to ASTER. Figure 11. Shaded relief showing misrepresentation of coastal shoreline and false islands in ASTER GDEM (left) compared to NGA DTED® (right) for a sample cell in Canada (at N72 28 15 W079 59 22). Water is blue. Figure 12. Shaded relief with an example of land form artifacts in Bosnia cell N44 E20 in the ASTER GDEM (left) that do not appear in the reference SRTM DTED® (right). Figure 13. Shaded relief of elevated linear artifacts in the ASTER GDEM (left) that may correspond to roads that don’t appear in the reference SRTM DTED® in Argentina (at S40 17 36 W066 34 21). Figure 14. Shaded relief of trenches in the ASTER GDEM (right) in Thailand that do not appear in the reference SRTM DTED® (left) (at N15 08 59 E100 24 37). Figure 15. Pits along ridgelines in the ASTER GDEM (left) compared to the reference SRTM DTED® (right) for Afghanistan cell N32 E066. Figure 16. Pits in the ASTER GDEM (left) compared to the reference SRTM DTED® (right) for Bosnia cell N44 E020. SUMMARY AND CONCLUSIONS This report summarizes NGA’s evaluation of a new ASTER Global DEM. The NGA analysis is based on a sample of 284 one-degree cells in 20 geographic areas and 16 countries, covering latitudes from 20° S to 80° N. The elevations in these areas ranged from sea level to over 6,700 meters with varying terrain relief. Two types of files were provided – the DEM in 1°x1° cells at a 30-meter post spacing and a “Numbers” file that gives the number of scenes (passes) used to compute the elevation for each pixel in the final ASTER product. The ASTER GDEM is referenced to WGS 84 and its associated EGM96 geoid so no conversion was applied. Aggregated statistics were computed for each of the 20 areas based on comparisons of the ASTER elevation data to reference data in the form of other DEMs and ground control points (GCPs). SRTM DTED® 2 was the primary reference DEM source, supplemented by other satellite and cartographic sources of DTED®. The GCPs were photogrammetrically-derived and were compared to both the ASTER and the reference DEM sources to assess the relative differences. Each geographic area was visually inspected to detect topographic anomalies and terrain artifacts that are not apparent in the aggregated statistics derived from millions of data points. To assess the accuracy of the ASTER GDEM, NGA computed and compared mean elevation differences between the ASTER GDEM and reference DEMs and between the ASTER GDEM and GCPs. NGA also noted differences in the minimum and maximum elevations in each area and the prevalence of spikes and wells in the terrain. The number of scenes used to compute the elevations was also considered in the analysis. In general, ASTER elevations in all areas are on the average biased negatively (i.e., lower) with respect to the reference DEMs, even after taking into account that SRTM elevations represent a reflective surface and are positively biased with respect to the bare earth when foliage is present. (It is also assumed that ASTER data are of a similar reflective surface nature and not adjusted to bare earth.) The mean ASTER elevations are lower than the GCPs in every area. The ASTER GDEM compares moderately well with the other DEM reference sources in the areas below 60°; but above 60° where there are no SRTM source data, specifically in Alaska, Russia and Canada, the ASTER GDEM quality deteriorates. In these areas, there are gross anomalies at single posts as well as over larger areas that have sizeable offsets compared to the reference data – as much as 3,000-4,700 m in some places. Although the 30-meter resolution ASTER GDEM was produced from 15-meter resolution VNIR measurements, the actual resolution of the DEM is noticeably lower than the equivalent 30-meter reference DEMs, as depicted in some of the shaded relief images discussed earlier in this report. Based on an examination of the elevation differences resulting from shifting the ASTER GDEM horizontally in 1-post increments over the reference DEM, there was no detectable improvement in the DEM differences that would indicate a consistent horizontal offset in the ASTER GDEM. However, it appears that the ground resolution of the ASTER GDEM is significantly less than the SRTM DTED®. Accounting for the mean elevation offset of the ASTER GDEM relative to the reference DEMs, the DEMs compared within a range of about 4-15 meters (90% vertical error), excluding Russia, Alaska and Canada, which were higher. When compared to the GCPs, the variability of the ASTER elevations is much greater than in the DEM-to-DEM comparisons, ranging from 10-25 meters, except for the Russia sites (25-34 m) and Canada (34 m). The comparable variability in the reference DEMs relative to the GCPs is 5-11 m, with two exceptions in Bolivia (15 m) and Alaska (24 m). In areas where there are other DEM sources besides SRTM to compare to the GCPs (e.g., Libya – SPOT 5, Bosnia and Iraq – photogrammetrically-derived DTED®), the error statistics are consistent with the SRTM DTED® results. These statistics indicate higher average noise and less precision in the ASTER GDEM than in the reference data (by a factor of 2), and this is confirmed by visual examination of the ASTER GDEM, as illustrated in some of the graphics and shaded relief images shown in earlier sections of this report. It was also possible to obtain a direct estimate of the vertical error of the ASTER GDEM in each area by assuming a maximum 10-m error for the GCPs and applying the computed ASTER-GCP difference error statistics to derive a 90% vertical error for ASTER. The estimated ASTER error ranged from 4-33 m and would be higher if a lower value were used for the estimated GCP error. Based on this measure, 12 of the 20 geographic areas would meet ASTER’s vertical accuracy specification of 20 m (95% confidence). It was clear from an examination of the ASTER “Numbers” file that there was great variability in the number of stacked scenes (passes) used to compute the elevation values. Some DEM areas are based on only one or two scenes (e.g., Russia, Australia, Kazakhstan) compared to other areas with 10 or more scenes that were averaged (e.g., Korea, Bolivia). Cells within the same geographic area can have significantly different coverage as well. In some cases, the higher vertical errors may be attributable to a scarcity of useable data (possibly Russia, for example); however, no generalizations could be made with the limited information and time NGA had available for this evaluation. (Note that two good highly accurate scenes may yield a better elevation value than 10 scenes with widely differing values of less accuracy.) All 20 geographic areas were visually assessed for their depiction of the terrain. A variety of significant data artifacts and anomalies were found in multiple areas. Mosaicking artifacts were common, in many instances related to incorporation of areas with different numbers of scenes into the final GDEM mosaic. These manifested themselves as ridged outlines that could be discerned throughout the data set. Magnitudes typically range from 5-20 meters. Alternate data sources used to fill voids in the ASTER GDEM were not smoothly inserted, leaving distinct physical boundaries and discontinuities between data sources. Large pits and wells are common in many areas as are spikes. It appears that no systematic filtering was done to detect and remove these. Since no attempt was made to identify, delineate and edit water bodies during the ASTER GDEM production, it is not surprising that the depiction of water bodies and coastlines is not good. The automated processing mistook water for land in some cases, created false islands, and generally was unable to distinguish water bodies from the surrounding terrain. As a result, inland water bodies are not readily discernible by inspection of the terrain. The use of the ASTER radiance measurements and a kilometer-resolution GTOPO30 water mask are inadequate for defining shorelines at a level commensurate with the 30-m ASTER GDEM resolution. Structural features in the terrain were observed in the ASTER GDEM that were totally absent in the reference DEMs. It is not clear how and why these were created in the ASTER data processing. Bad data were observed above 60° N latitude in Alaska, Canada and Russia. Relatively large regions have vertical errors with respect to the reference DEMs of one kilometer or more. Some of these areas have been voided out in the public release version of the data, but there are still DEM differences of several thousand meters with respect to the reference DEMs. RECOMMENDATIONS Overall, the quality of the ASTER GDEM appears to be less than the SRTM DTED® 2. Although the aggregated error statistics are not that bad in many areas, the data in its present form would be very difficult to trust and use without careful review and editing. Within the SRTM coverage area (56° S to 60° N), the ASTER data may be useful for filling voids in the SRTM DTED® on a case-by-case basis after a thorough examination. Above 60° N where no SRTM DTED® are available and where other DEM data are scarce, ASTER has problems. There are obviously ASTER scenes available at these latitudes, but the data need to be reprocessed to remove the major anomalies and artifacts that are present. Even then, one would need to examine an area’s data for inconsistencies before using them. The lack of water body depiction and the frequency of artifacts would require extensive editing and filtering of the ASTER GDEM to convert it to a DTED® product. At best, the ASTER GDEM can be used as an independent source of elevation data for areas where no other reliable data exist, and could be used on a case-by-case basis to support specific applications. REFERENCES Farr, T. G. et al., The Shuttle Radar Topography Mission, Reviews of Geophysics, 45, RG2004, 2007, 33 pp. NASA Jet Propulsion Laboratory, URL: asterweb.jpl.nasa.gov, 2009. NIMA 2000. Performance Specification, Digital Terrain Elevation Data (DTED®), MIL-PRF-89020B, 23 May 2000. Slater, J. A. et al., The SRTM Data “Finishing” Process and Products, Photogrammetric Engineering & Remote Sensing, Vol. 72, No. 3, Mar. 2006, pp. 237-247. Appendix. Frequency Distributions of the Absolute Values of the ASTER-GCP Differences and Reference DEM-GCP Differences Figure A-1a. Distribution of |Aster-GCP| for Alaska. Figure A-1c. Distribution of |Aster-GCP| for Argentina. Figure A-1e. Distribution of |Aster-GCP| for Bolivia. Figure A-1b. Distribution of |NED-GCP| for Alaska. Figure A-1d. Distribution of |SRTM-GCP| for Argentina. Figure A-1f. Distribution of |SRTM-GCP| for Bolivia. Figure A-1g. Distribution of |Aster-GCP| for Bosnia. Figure A-1i. Distribution of |Aster-GCP| for China1. Figure A-1k. Distribution of |Aster-GCP| for China2. Figure A-1h. Distribution of |DTED-GCP| for Bosnia. Figure A-1j. Distribution of |SRTM-GCP| for China1. Figure A-1l. Distribution of |SRTM-GCP| for China2. Figure A-1m. Distribution of |Aster-GCP| for Kazakhstan. Figure A-1o. Distribution of |Aster-GCP| for Nigeria. Figure A-1n. Distribution of |SRTM-GCP| for Kazakhstan. Figure A-1p. Distribution of |SRTM-GCP| for Nigeria.