NASA Technical Memorandum 104594, Vol. 4 MODIS Technical Report Series Volume 4, MODIS Data Access User's Guide - Scan Cube Format Virginia L. Kalb Goddard Space Flight Center Greenbelt, Maryland Thomas E. Goff Research and Data Systems Corporation Greenbelt, Maryland National Aeronautics and Space Administration Scientific and Technical Information Program 1994 https://ntrs.nasa.gov/search.jsp?R=19950010172 2020-06-16T08:49:55+00:00Z This publication is available from the NASA Center for AeroSpace Information, 800 Elkridge Landing Road, Linthicum Heights, MD 21090-2934, (301) 621-0390. Table of Contents INTRODUCTION BACKGROUND IDENTIFICATION SCOPE PURPOSE AND OBJECTIVES DOCUMENTATION CONVENTIONS TOOLKIT USAGE QUICK EXAMPLE TECHNICAL POINTS FUNCTIONAL OVERVIEW PSEUDO CODE EXAMPLE FUNCTION CALL SEQUENCE ADVANCED EXAMPLE FUNCTION REFERENCES LIBRARY ACCESS AND INSTALLATION APPENDICES Appendix A - MODIS Spatial Domain Descriptions The MODIS Scan Cube Spatial Domain The MODIS Rectilinear Domain The MODIS Mapped Domain Appendix B - Data Product header include file description Appendix C - The DataDescriptors.h Library Include File Description Appendix D - Data Product Header example Appendix E - MODIS Dataset Internal Format DATASET contents example Appendix F - Library Dataset Interactions GLOSSARY REFERENCES 1 1 2 3 3 5 6 7 8 11 11 12 14 17 30 31 31 31 33 33 35 37 39 45 48 50 52 58 List of Figures Figure Figure Figure Figure Figure Figure Figure Figure Figure Figure Figure Figure 1 2 3 4 5 6 7 8 9 10 11 12 Simple Algorithm Example Simple Algorithm make File Detector Numbering Pseudo Code for a Minimum Algorithm Multiple Scan Cube Processing Illustration Two Dimensional Access Example MODIS Level-1 Simplified Scan Cube Overall Dataset Contents Memory Allocation for Variable Coincident Spatial Sizes Dataset Contents Example Data Flow Illustration Algorithm Dataset Interactions 7 8 10 11 14 16 31 45 46 48 5O 51 List of Tables Table 1 Table 2 Table 3 Table 4 Documentation Conventions List of Files MODIS Level-lB Data Product Scan Cube Sizes Header Parameter Names 5 30 32 47 /v INTRODUCTION The library functions described in this document allow read and write access to the Earth Observing System (EOS) Moderate Resolution Spectroradiometer (MODIS) derived data products. They also provide access to remotely sensed data from other scanning instruments, both heritage and new instruments, that have been transformed into the MODIS scan cube format. These functions have been designed to allow all users to have easy access to all information components contained in MODIS formatted datasets. They also provide a flexible and simple method for writing MODIS formatted datasets. The MODIS scanning technique is unique when compared with other instruments, due to its large off nadir viewing angle coupled with the simultaneous capture of multiple lines. Nevertheless, MODIS instrument data will be available as both raw detector counts and at-satellite radiances in the instrument viewing geometry. This includes all aberrations inherent in the MODIS scanning technique. Previous instruments with similar types of instrument viewing distortions have been resampled before general consumption and subsequent ingestion by science algorithms. However, highest algorithmic accuracy for science data products is maintained by using this unresampled MODIS data. The high MODIS data volume (approaching a half terabyte per day) has been a driving factor in the library design. To avoid recopying data for user access, all I/O is performed through dynamic "C" pointer manipulation. The user has easy access to any instrument band or data product parameter in a dataset through the "C" data structure syntax. This technique is also used for creating output data products. BACKGROUND MODIS instrument data is used by a wide array of scientists to derive information about the Earth. In general, scientists ingest MODIS data along with ancillary data to produce MODIS Data Products. These products are then used as part of the knowledge base of information to determine long term changes in the Earth biosphere, which is then used to make policy decisions at all governmental levels. MODIS algorithms are executed in, and the resulting data products are archived by, the production EOS core system (ECS) distributed information system (EOSDIS) which also archives the resulting data products. Data derived from the MODIS instrument can be divided into three types, corresponding to three spatial domains: the scan cube domain, an orbital rectilinear domain, and a geographically based mapped domain. The scan cube domain consists of all Data Products that are produced at MODIS instrument pixel IFOVs, or spatial aggregations thereof. (See Figure7 in Appendix A for a visualization of the MODIS scan cube.) These products are the MODIS Level-1A raw counts, MODIS Level-1A Geolocation Data Product, MODIS Level-1B at-satellite radiances, and Level-2 products that are produced in the true MODIS scan geometry. The rectilinear domain consists of MODIS scan cube data that has been resampled into a coordinate system on EOS across track and along track axes. The mapped domain encompasses all data products that have been gridded, binned, or otherwise aggregated onto a map projection or other Earth geoid based coordinate system such as the MODIS Level-3 data products. Algorithm developers must consider the spatial domain in which the algorithm will operate. Algorithms that do not incorporate a spatial component will operate in the scan cube domain. Algorithms that incorporate spatial components or will use MISR Level -lb2 or similar data for Level-2 scan cube processing should consider processing in the rectilinear or mapped domains. Algorithms that depend on more accurate band to band registration or uniform ground field of views should also consider the rectilinear and/or mapped domains. Algorithms that depend on in situ, external ancillary inputs, other satellite or instrument data sources, or require fixed ground areas will require interactions with data in the mapped domain. Algorithm developers must be cognizant of the MODIS "bow tie" overlap effect in the scan cube domain. See the various MODIS publications listed in the reference section of this document for "bow tie" effect descriptions. In all cases, the user needs to consider the possibility of modifying the algorithm to use the "earlier" domains when possible. Transforming data products directly from the scan cube domain to the mapped domain, without going through an intermediate rectilinear resampling, is highly recommended. This will minimize data resampling, cross product data dependencies, _d computer based processing power. It should be noted that the historical Landsat TM data and the EOS MISR data are resampled from the instrument geometry to scenes or orbital axes before any science algorithms are applied. More complete descriptions of the MODIS spatial domains are given in Appendix A to this document. Considerations for each domain, and transformations among domains are given in the "MODIS Processing Spatial Domains" paper (see Reference section). IDENTIFICATION This document is the first in a series of User's Guides describing the library and utility functions that the SDST will be generating. This edition of the MODIS SDST Library USER'S GUIDE applies to the September 1994 delivery of the MODIS scan cube I/O access routines, version 1.0, library source code. It includes descriptions of all the pieces necessary to use the library, including sample data product header include files for AVHRR. Additional data product header include files for actual MODIS products and other data source header include files will be available via electronic means and are not a part of this edition of the guide. Several small test datasets derived from AVHRR are included in the distribution and used for illustration purposes in this document. More complete datasets are available from the MODIS SDST or can be created using information in the appendices to this document. This document is written and maintained by members of the MODIS Science Data and Support Team under the auspices of NASA Goddard Space Flight Center, code 920.2, in Greenbelt, MD. SCOPE This guide contains information applicable to both the use and maintenance of the library functions. It is directed toward four classes of users: normal MODIS algorithm creators and data users, sophisticated creators of spatially derived algorithms, those who will provide additional instrument datasets (other than MODIS), and those who will alter existing or add new data product header include files. This document does not describe the internal workings of the library source code, but provides descriptions of the methods utilized by the library to access the datasets to aid users in understanding the library's capabilities. This edition of the MODIS SDST Library User's Guide describes the initial "C" language version of the SDST Library input and output (I/O) functions. These were created for the purpose of reading and writing currently available data formatted as MODIS data products. A future release of this document will include similar FORTRAN functions written in either FORTRAN77 or FORTRAN90. The current suite of functions at this release is limited to the MODIS Level-1A, Level-1B, and Level-2 Data Products. These are the data products that are a part of the scan cube domain. It does not cover access to Level-3 or Level-4 data products which are in the mapped domain. Use of this library relies heavily on the user's understanding of how to use each "C" data structure as defined uniquely for each format Data Product. These are defined in data product header include files which are unique to each formal Data Product. These header include f'des, available to all users of this library, will be under SDST configuration management and distribution. They will be published and announced via Email to all MODIS algorithm developers. This will help to insure the synchronization of data product definitions among the MODIS Team Member responsible for the formal Data Product, the Science Product Support Office (SPSO) database entries, the data product header include files associated with this library, and all users of the data products. The MODIS SDST proposes to write these data product header include files for algorithm developers in conjunction with the SPSO and MODIS Team Members or their programming representatives. Although the access to MODIS data is easily understood, the programming techniques utilized in the header include files are necessarily complex to accomplish the encapsulation of the data products. Documentation of these techniques is given for completeness but is not mandatory reading for library use. PURPOSE AND OBJECTIVES This library has been designed to allow read and write access to data from a suite of scanning type instruments whose data have been transformed into a MODIS viewing geometry and written in a MODIS scan cube data structure. This allows a common library to be used for algorithm development. Simulated and/or test data sets will be used for algorithm development and production environment testing before actual MODIS data are available. The currently available datasets emphasize AVHRR data, with TM data forthcoming, but are expected to be expanded to other instruments such as MSS, CZCS, MAS, or other data sources on a user requested basis. The internal format for the dataset contents and a simple example are given in Appendix E to this document. TheMODIS scancubeconceptcontainsdatavaluesin athreedimensionalraggedarrayformatwherethe arraydimensionscorrespondto theacrosstrackdirection,thealongtrackdimension,andthenumberof bands.Theraggedarrayterminologyappliesto adatastructurein whichthesizesof eachcomponentof thearrayarenotconstant.Forexample,atwo dimensionalragged array data structure containing the words of a sentence would use the number of words in the sentence as one dimension and the variable number of characters per word in the other dimension. In the MODIS case, the number of bands is constant, but the number of lines per scan cube in the along track dimension is a function of the band number, and the number of pixels across track varies across scan cubes. The three dimensional scan cube array can be collapsed into a two dimensional data array consisting of across track pixels and bands by setting the along track dimension to one (1). This allows single line scanning instruments to be accommodated by the MODIS scan cube data structure. The number of elements in each dimension are included in the library structures and are explicitly available (no function call required) to the user. This allows the use of these library functions for uses beyond the scope of the MODIS effort. The library implementation, which incorporates a "data product header include file" technique, allows user written algorithms to determine the sizes of all data array elements as a function of the instrument data source. This allows a single algorithm process (program) to be easily written in a manner such that any instrument source can be ingested. Information such as the number of bands, band central wavelengths, band widths, number of pixels, etc. can be made available from each data product header include file. Multiple data product header include files can be used within a single program, thereby allowing simultaneous access to other data products. An example of this facility comes from AVHRR in which the data has a different number of across track pixels and bands from MODIS, and from TM in which a different instrument IFOV for each band is employed. AVHRR data are currently made available to algorithm developers in raw instrument counts, the original NOAA albedo / brightness temperature data values, and also convened to radiance energy values. Actual MODIS data will be available only in radiance energy values. Utilizing the library header include files, which contain data type and structure specifications for each of the available input and output data source, allows the data types to be known to an application program without performing data type conversion within the library. Additional data types can be accommodated with a custom header file describing the new data source. The library does not need to be recompiled to add new data sources. The function calling sequences and parameters are expected to remain constant even though the underlying implementation may change. This ensures a stable interface for algorithm developers. The remainder of this document addresses the I/O routines and illustrates, by example, how to access the data in the MODIS scan cube. Each section of this library contains functions that allow a science algorithm to input and output MODIS data products, or other instrument data masquerading as MODIS scan cubes. Library functions that transform or resample data products from MODIS scan cubes to other forms have yet to be defined, and are not a part of this library effort. 4 DOCUMENTATION CONVENTIONS The conventions used in this document are described as follows: Table 1. Conventions Used Times Roman font Courier font Arial carots ( < and > ) double quotes ( " ) Leading Caps UNIX meta-characters Normal text and narrative comments added to source listings. Program source code source listings copied from the library distribution and pseudo code examples Program output as if received at a terminal Enclose generic items to be replaced by specific user supplied names or descriptions. This convention is also used in UNIX source listings to specify the default location for operating system library include files. Enclose a character string set that is to be considered as a unit. Are used to designate a formal noun, as in a reference to a formal registered (with the SPSO) Data Product. A method of designating occurrences of characters using the UNIX conventions for metacharacters, specifically the * character to represent one or more occurrences of a character. TOOLKIT USAGE Four methods for illustrating the use of the library functions, and a section with technical information, are included in this section. The _ _ presents an actual small algorithm with three parameters. The source code is included in the library distribution and can be quickly cloned to generate simple algorithms which can be expanded with time. This is the easiest method for getting started with the library function calls. The Functional _ section goes into the philosophical aspects in the design and use of the I/O library. It includes descriptions of the functions and describes the sequence and coupling among the functions. The Advanced _ section presents a sample algorithm that exercises the library's capabilities for handling multiple scan cubes at a time with the use of moving windows. This allows two dimensional access across the boundaries of these scan cubes. This example illustrates the library's use with algorithms which need to know the relative locations of neighboring pixels for spatial operations. The final Function Reference section lists the functions in alphabetical order by function entry point names and included the details about each function call, such as error returns and ANSI function prototype definitions. The included Technical Points section presents a list of assumptions and notes concerning the use of these I/O library functions. 6 QUICK EXAMPLE To illustrate the simplicity of library usage, a fully functional actual code example with make file is included in this section. It reads MODIS Level- 1B radiances (actually AVHRR in MODIS form) and calculates a simple vegetation index (VI) with an average thermal temperature and a quality assurance (QA) value as additional parameters. Only those lines of source code directly applicable to a potential science algorithm are commented. #include #include #include "AVHRR.h" #include "VI.h" #include "MOD_IO_prototypes.h" <- define AVH data in MODIS format <- define pseudo VI output data product main() { AVHRRscancube ibuf; VIscancube obuf; short fdin, fdout,ierr,k; float pixChl, pixCh2, pixCh4, long bufid; long npixels, nx, ny, x, y; <- declare an input data structure <- declare an output data structure pixCh5; fdin=MOD_IO_openMasterInputDataset("AVHRR_Orbit543.Llb.MODISlike", &AVHRRheader); fdout = MOD_IO_openOutputDataset("viOutputfile", &VIheader); while ((bufid = MOD_IO readSwath(fdin, &ibuf)) > 0) { <- inputthedam ierr = MOD IO allocateOutputBuffer(fdout, &obuf); <- ob_/nanou_utbuffer npixels = ibuf.b[0].nPixels; for (k:0; k i. 0)obuf.viQuality.data[k] = +i. 0; obuf.avgTemp.data [k] = (pixCh4 + pixCh5) /2. 0; <- fmd_eaveragetempemmm } ierr = MOD_IO_writeSwath (fdout) ; <- writetheou_utbuffer } MOD IO closeDatasets (); Figure 1. Simple algorithm example. 7 The threeVI parameters (vegIndex, viQuality, and avgTemp) are treatedas subelemen_ of a complex data structure. Each element of this structure corresponds to one instance of a complex data product value. Use of the "C" data structure concept instead of separate arrays guarantees the synchronization of the various science parameters. The above example can be compiled with the MODIS I/O Utility Library via the following UNIX make f'tle listed in Figure 2. This is a minimalist make file and does not include directory details and environmental variables. See the make fdes, README file and INSTALL file in the library distribution for details. (Note that the "-Aa" option flag is the ANSI switch on HP and SGI, not Sun compilers.) VI : vi. o MOD_IO_lib. o cc -Aa -o VI vi. o MOD_IO_lib. o vi.o: vi.c MOD IO lib.h AVHRR.h VI.h DataDescriptor.h cc -Aa -c vi.c MOD IO lib.h DataDescriptor.h AVHRR.h VI.h MOD_IO_lib.o: MOD_IO_lib.c MOD_IO_lib.h DataDescriptor.h cc -Aa -c MOD_IO_lib.c Figure 2. Simple example make file. The simplicity of this source code is made possible by the predef'med data product header include file. The header file for the example VI data product that defined the output data product and the parameters of this output data product is included in the library distribution. The explanation of the contents of this header include file is the subject of a Appendix D to this document. TECHNICAL POINTS This initial release includes "C" bindings only. Bindings for FORTRAN functions can not be implemented due to the lack of data structures in FORTRAN77. Bindings for FORTRAN90, however, are possible. Algorithms that are written in FORTRAN77 (with POSIX and ECS approved extensions) will require the writing of "wrapper" functions that extract MODIS data for each band or parameter individually and supply this data to a calling FORTRAN function. The SDST is considering adding this capability and needs user input to generate a requirement. • Each band or parameter in a scan cube in the data set can be accessed as either a linear array (vector) or a two dimensional array (matrix). This is the user's choice when writing the algorithm. Scan cube sizes can vary from one scan cube to the next. The bands or parameters can have different array sizes that are integer multiples of each other to accommodate multiple data resolutions. 8 • Outputdataproductscancubesarecoupledto amasterinputscancube.Thisallows the MODIS scan cube Geolocation Data Product to be applied to an output product, by reference. That is: a new geolocation data product does not need to be created for each output product, the input geolocation applies to the output product as well. This coupling is maintained in all subsequent generations of MODIS Data Products. Additional input data in the scan cube or rectilinear domain can be accessed as ancillary inputs by this same set of library calls. No additional functions are needed, but care must be exercised in the scan cube domain to maintain spatial registration. The onus is on the user, not this utility library, to insure proper usage of ancillary data. • All accesses to MODIS data are assumed to be sequential. Scan cube data can not be accessed in a random manner with these library calls. • Error returns are returned to the user as function returns. Negative numbers are an error, zero or optional positive numbers indicate a non error condition. Negative error returns from -1 to -99 are non fatal (soft) errors such as encountering an end of file (EOF) condition. Error returns from -100 to -??? indicate a fatal (hard) error such as a memory overflow or illegal dataset. This utility library will endeavor use the generic PGS error determination routines in a future release, after they are more fully understood by the developers of this library. NOTE that this current library release (1.0) produces error messages to stderr and terminates if a fatal error occurs. This is a temporary safety mechanism for algorithm development and will not be allowed in the PGS environment. Error return values and messages are documented in the Alphabetized Function Reference section of this user's guide. • All functions use ANSI prototyping beginning with the 1.0 release. These prototypes are available in a "C" header include file, "MOD_IO_prototypes.h", included with the library distribution. A set of headers will be publicly available that define the Data Product data structures, file set descriptions, constant variables (typedefs), etc. Any header include files for formal Data Product structures generated by algorithm developers are required to be given back to the SDST Utility Library configured management access point for availability to all library users. A example generic header is described and included in Appendix B to this document. • All function calls dynamically allocate space within the library and set pointers to that data space in the user's data structure. This frees the science algorithm developer (the user of the library) from concerns about memory allocation details and data array boundary overruns (memory leaks), provided the header structures are used. Most importantly, it allows variable length scan cube records for both the science and calibration portions of each MODIS scan. This ability applies to both the scan cube and rectilinear domains. • The scan cubes are assumed to be sequential in time. That is: no global descriptor such as a scan cube counter or equivalent time based discriminator is implemented, but will be required to identify scan cubes in the production environment. Opening more than one input dataset, (granule or orbit), is allowed. Additional datasets that do not match the global dataset descriptor or master scan cube ID will product a non fatal error in future releases. The user, not the library, is then responsible for mapping or "stitching" techniques to insure geographical registration. 9 More thanonescancubecanbeaccessed(sequentially)for eachopeneddataset.Thelimit to thenumber of inputdatasetsconcurrentlyopenis three(3),with useraccessto five (5) scancubesperdataset.This accesscanbe"roundrobin'ed"within thefive (5)at atimelimit. Three(3)outputdatasetscanalsobe generatedconcurrently.Theselimits areparameterizedin thelibrary internalheaderfile andcouldbe easilychangedby recompilingthelibrarywith differentvalues.Notethatlargernumberswill increasethe libraryinternaltablesizes. Usersinterestedin theMODIScalibrationandcorrelationwith documentationsuppliedby SantaBarbara ResearchCenter(SBRC, theMODIS instrumentbuilder)will needtheinformationin thisparagraph. Sciencealgorithmdevelopersshouldnotneedthis informationbut it is includedherefor reference purposes.Externaldocumentationwith thedetectornumberingschemeusedby SBRCwill requirethat theuserof MODIS Level-1AandLevel-1Bdataknow aboutthe"flipping" of thisdetectornumbering with respecto thescancubelinenumberingsequence.SeeFigure3.TheUtility Libraryusesa"line within scancube"numberingschemethatincreasesin thealongtrackdirection.This is oppositeto the SBRCdetectornumberingscheme.This line numberingschemeis usedin theMODISLevel-1A, Level-1B,andLevel-2DataProducts. J_ o t- O l detector 10swathdetector 1 scan cube l- across track SBRC Detectors Figure 3. dl< m =canUne0 swath =man line9 scan cube m across track SDST Scan Lines Detector numbering. w m ._c E o • The input and output MODIS data calls can be intermixed within the same program, but must be kept in the order illustrated in the included source code and pseudo code examples. This sequencing facilitates synchronization of the Geolocation Data Product with all output Data Products. • The library is designed to enforce function call ordering and will generate error conditions otherwise. • Fully commented sample programs that include the incorporation of heritage and/or dummy algorithms are available in the library distribution or from members of the SDST algorithm transfer team (ATI') directly. Additional help, information, and sample code is available from ATT members. 10 FUNCTIONAL OVERVIEW This utility library can be used in a simple manner in which a MODIS Data Product is generated from only the MODIS Radiance input values, or a more complicated manner in which concurrent use of multiple scan cubes and more than one input source are used to generate a MODIS Data Product. The example pseudo code, listed in Figure 4, introduces the simple techniques for accessing MODIS data. A discussion of more complex capabilities of this library is contained is a later section of this user's guide. PSEUDO CODE EXAMPLE The following pseudo code example contains an ordered list of the basic function calls required to develop an algorithm. This sequence of calls performs the following functions. The MODIS scan cube domain input Data Products are obtained with a sequence of input library calls, allocation of output scan cubes are obtained via a library call, the algorithm results are placed into the new scan cubes, and Data Products are transferred to the f'de system with output library calls. The library is initialized with data set "file open" calls, scan cubes are staged with a "read cube" call, individual MODIS bands or other data product parameters are made available to the user within a "C" data structure, output scan cubes are allocated, output data product parameters are calculated, the scan cube of data product is placed into the output file, and finally the data sets are closed with a "file close" call. Multiple bands or parameters are accessible by accessing elements of each input data structure, and multiple sequential scan cubes are obtained with multiple "read cube" calls into separate input data address areas. Pseudo code with the library function names for a minimum algorithm is shown in Figure 4. filein = MOD_IO_openMasterInputDataset( inputFileName, InDataDescriptorAddr ) fileout = MOD_IO_openOutputDataset ( outputFileName, OutDataDescript orAddr ) for every scan cube: MOD_IO_readSwath ( filein, InBufferAddr ) MOD IO_allocateOutputBuffer( fileout, OutBufferAddr ) npixels = InBufferAddr.bandl.nPixels - 1 conloute the parameters for this algorithm, for k=0 to npixels OutBuffer.paraml.data[k] = a function of( InBuffer.bandl.data[k], etc. ) MOD_IO_writeSwath( fileout ) end of for every scan cube MOD IO closeDatasets () Figure 4. Pseudo code with the library function names for a minimum algorithm. Note the use of the "C" data structure OutBuffer. paraml, data [k] in which each derived value of an output Data Product is placed into an output buffer ' OutBuf fer', in a specified parameter ' pararal ', in a linear array ' data [ ] '. The size of the array ' data [ ] ', and the corresponding range of the index ' k', are dynamically altered by the value of ' npixels ' for each read of a scan cube. This is necessary to prevent the placing of values outside the memory bounds of the output data product. 11 LFUNCTION CALL SEQUENCE The library functions must be invoked in the proper order as illustrated in Figure 4 - Pseudo Code example (previous section). The functions performed by the individual function calls, and their coupling with other functions, are given in the following paragraphs. The 'filein - MOD IO_openMasterInputDataset( inputFileName, InDataDescriptorAddr ) ' functionreturnsa unique identifier('filein' )toa datasetcontaining the MODIS Level-1A, MODIS Level-1B, MODIS Geolocation, or other MODIS Level-2 input Data Products. It requires the user to specify a character array 'inputF£ 1 eName' containing the fully qualified name of the MODIS or "MODIS like" input data files. The library checks the input dataset for a predefined dataset type specifier e.g. "MODIS L1B". The user supplied parameter 'InDataDescriptorAddr' is the address of an instance of a data structure defined in the unique product header include file for this input product. The formal name of this data structure is given in that header include f'fle. The header include file also contains a character array with a dataset contents specifier e.g., "AVHRR_albedo_and radiances". These characters are used as a sanity check with the dataset embedded header information to make sure the user knows the source and type of data expected. For example, an AVHRR dataset can be used as input to an algorithm for development purposes, but the user must know apriori (and specified in the AVHRR.h include file) which AVHRR bands to use in the algorithm. These bands will have different central wave numbers and band widths for other instrument data sources such as MAS or MSS. Additional information, unique to each dataset, is available to the library through the dataset contents. The header include files contain the generic dataset parameters and the specific formatting information for that dataset. The' fileout - MOD_IO_openOutl)utDataset( ol_utputFileName, OutDataDescriptorAddr ) ' funcfon returnsa unique identifier('fileout ')toa datasetcreated by the library to contain the user derived output Data Product. It requires the user to specify a character array 'outputFi 1 eName' containing the fully qualified name of the MODIS or "MODIS like" output data file. The library places the predefined dataset type specifier CMODIS") into the output data product volume header. The user supplied parameter 'OutDataDescriptorAddr' is the address of a data structure defined in the unique product header include file for this output product. The library also places a character array with a dataset contents specifier e.g. "AVHRR_derived_Locust__probability" into the output file volume header. This is obtained from the data product formal "" file. Geolocation that has been tagged to the Master input scan cubes also applies to the output data product. The master input scan cube dataset must be open before this cal can be made. The 'err - MOD_IO_readSwath( filein, InBufferAddr ) ' functionreadsa scan cube's worth of data from the dataset specified by ' fiIein' into the library space, not the user space. The address of the user data structure parameter ' InBuf ferAddr ' is passed to the library function. The library passes pointers to the science data to the user through the user's data structure ' InBu f f er'. In addition, the library places the number of Earth viewing along track elements, the number of across track elements per line, and the number of total spatial elements (across track times along track) into this user supplied pointer to the scan cube data structure. For the real MODIS instrument, the nominal number of across track spatial elements is 1354 with a fixed 10 detectors along track, thereby creating 13540 spatial elements. MODIS bands 1 through 7 will have larger numbers for these values. (Note that spatial elements are not the same as IFOVs for MODIS bands 1 through 7). For the AVHRR MODIS "look alike", these 12 returned numbers are 2048 pixels across, 10 detectors down, and 20480 total elements. Caution: this returned value can be different for each MODIS scan cube because the number of pixels across a MODIS scan is ground commandable. This can also vary across instrument data sources. The library handles data sources with a different number of pixels in each of the two dimensions, and a different number of pixels per band (or parameter), and can handle these within the same algorithm program. Each input Data Product is represented as a data structure to the user. Pointers are used within the data structure to allow access to each individual parameter. This technique allows the library to allocate memory space dynamically when the size of the scan cube changes. This is performed without user intervention. It also involves no memory transfers and is thus very efficient. The user has access to all the information and data sizes required to use the input data via a set of predefined (in the data product header include file) data structure elements. This is explained in the header f'de descriptions in a following section and also illustrated in examples included in the appendix to this document. IMPORTANT: Users who plan to use a scan cube call for spatial processing need to be knowledgeable of the detector numbering, band to / from detector translations, band registration over variable spatial IFOVs, and pixel line numbering schemes. ff the input file is a MODIS Level- 1A dataset, additional information as defined in the MODIS Level- 1A Data Product header include file is available. This allows access to the onboard calibration data. Details on how to access these components is contained in the Level- 1A Data Product header include file and is not included here. The 'err = MOD IO_allocateOutputBuffer( fileout, OutBufferAddr ) ' function creates space in the library allocated memory area for an output scan cube that corresponds to the currently specified "Master scan cube". This synchronizes the output Data Product with the Master input data Product, thereby allowing the appropriate geolocation dataset to apply to this output Data Product. The output Fde is designated by the user supplied ' f 5.1 eout' indicator. All information that the algorithm requires (across and along sizes, number of pixels, etc.) is now available as structure elements in the formal output Data Product data structure pointed to by the user supplied address ' OutBuf ferAddr' These sizes must be used in the algorithm code to prevent 'stepping over' or overwriting the neighboring data elements. Extreme cases will produce a memory protection fault. The function prints an error message if something goes wrong. The ' MOD_IO_writeSwath ( fileout; ) ' function writes the user generated data from the library memory space onto the disk. The user parameter ' f i 1 eout ' indicates to the library which output dataset is to be written. An error code is returned if unsuccessful. The 'MOD_I0_closeDatasets () ' call,the 'MOD_IO_closeInputDataset ( filein ) ' call, and the ' MOD_IO_closeOutPutDataset ( fileout ) ' call deallocate the library memory space for either the specified input file, output file, or all dataset files. It also appends a trailer record to an output dataset to indicate a logical end of file (EOF). 13 ADVANCED EXAMPLE This section illustrates the use of the library for spatial processing by allowing more than one scan cube to be in memory at a time and utilizes the two dimensional access to the scan cube data. This code example performs a moving window spatial average over a 3 by 3 pixel area across scan cube boundaries. This requires the use of two input scan cube buffers for each output buffer. Figure 5 illustrates the spatial relationship for the indices in the code example. The X in the figure corresponds to the placement of the calculated average in the output data product. T | g j-direction Ior the next to lastcolumn, J ! RADIUS. in the j - dlrecUon= 2 ibuf[bsav] IIII ibuf[bufindex] for _ last column, / RADIUS. in lhej - direction = 1 L 2 Figure 5. Multiple scan cube processing illustration. The key to obtaining the data within the scan cube is the use of "C" data structures with dynamic pointers. The example code in this section is graphically illustrated in Figure 5. The data structure contents, including the pointers, is reinifialized on each invocation of a MOD_IO_readSwath or MOD_IO_al locateOutputBuf fer function. The data structure is declared by the user, but its address is passed to the library so that the library can manage dynamic memory allocation and fill the data structure. The user gains access to the data indirectly through the pointer mechanism. Note that while the data product header include f'de provides both one dimensional and two dimensional access to the same scan cube data, a single algorithm can not access the same data utilizing both techniques at the same time. #include #include #include "AVHRR. h" <- theMODIS headerincludefiles #include "MOD_IO_orototypes.h" #define RADIUS 3 #define Min(x,y) (x < y ? x : y) main() ( AVHRR2Dscancube ibuf[2]; AVHRR2Dscancube obuf; short fdin, fdout, ierr, <- bandindex; declare 2 input buffers of type AVHRR2Dscancube and one output buffer of the same type Figure 6. Two dimensional access example. 14 long bufid[2]; short bufindex, bsav, long noutput, nj, ni; float output; j, i, kj, ki, radiusi, radiusj; fdin=MOD_IO_openMasterInputDataset("scancubes.small", &AVHRRheader); fdout = MOD_IO_openOutputDataset("avgOutputfile", &AVHRRheader); if((bufid[0] = MOD_IO_readSwath(fdin, &ibuf[0])) < 0) exit(-l); bsav=0; bufindex=l; while((bufid[bufindex] = MOD IO readSwath(fdin, &ibuf[bufindex])) > 0) { ^^--Thiswhileloop_executedun_ereadSwathfuncfionm_catesnomomdamamavailab_. fprintf(stderr,"main: read id = %d\n", bufid[bufindex]); ierr = MOD_IO_allocateOutputBuffer(fdout, &obuf); <- requ_tanou_utbufferfrom _e_bmry. for(bandindex=0; bandindexxy[j+kj][i+ki]; noutput++; } ) ^^ sum up the pkel v_u_ in a 3 x 3 p_el area. The dam structure used hem addres_s each p_el by _l_ting the buffer _v] from an array of input buffe_ _ulTomv], indexed by the band number _andindex], further indexed m the science dam .dam w_ch poinu to the two dime_ion_army _presenmfion xy[ ] [ ], _ing normally computed army indices j+kj and i+_. obuf. b [bandindex]. data->xy [j] [i] = (output/noutput+. 5) ; ) ^^ the ou_ut product _ contained wi_ a single buffer obuf, _ing the same dam structure technique • e input buffer. /* Next to last line in the current scancube: need the top line from the next scancube: */ output = 0; noutput = 0; for (kj=0; output += output += output += ] kjxy [j+kj ][ni-2 ]; noutput++; ibuf [bsav] .b [bandindex] .data->xy [j+kj ][ni- 1 ]; noutput++; ibuf [bufindex] .b [bandindex] .data->xy [j+kj ] [0]; noutput++; ^ ^ NOTE: the ibuf indices bsav and bufmdex indicate that different scan cubes are accessed here. Figure 6. Two dimensional access example (continued). 15 obuf. b [bandindex]. data->xy [j ] [ni-2 ] = (output/noutput+. 5) ; /* Last line in the current scancube: need the top 2 lines from the next scancube: */ output = 0; noutput = 0; for (kj=0; kjxy[j+kj ] [ni-l] ; noutput++; output += ibuf[bufindex] .b[bandindex] .data->xy[j+kj] [0]; noutput++; output += ibuf [bufindex] .b [bandindex] .data->xy [j+kj ] [i] ; noutput++; ) "^ one scan cube from bsav and two scan cubes from bufindex. obuf.b [bandindex]. data->xy [j ] [ni-l] = (output/noutput+. 5) ; <- .5isused m round to integer numbers properly. ) bsav = bufindex; bufindex ^= I; /* we are using only 2 scan cubes so used to toggle the index */ ierr = MOD_IO_writeSwath (fdout ); } MDD IO closeDatasets () ; an exclusive or is Figure 6. Two dimensional access example (continued). The above example does not include the use of more than one ancillary input MODIS data product, or auxiliary data from non MODIS sources. Ancillary data, obtained from other data sources, can be accessed via these library calls and included in the algorithm code after conversion to MODIS scan cube format. Auxiliary data is obtained via function calls that am not a part of this library and can be in any data format. The algorithm writer must insure that these extra data sources am used appropriately by checking geolocation information, spatial sizes, units, and etc. associated with these ancillary data. This information is made available to the user via the dataset header contents and accessed via the data structure pointer mechanism. MODIS output data products am kept in synchronization with the Master input data product by this library. The auxiliary data function calls can provide correct data by examining the input parameters that specify the area and units requested by the calling program. 16 FUNCTION REFERENCES This section contains the alphabetical ordering of the function calls documented in this User's Guide. It is meant to be a quick reference to the individual function calls. Details of the ANSI prototyping specifications are included. All error messages produced by the library functions are documented here. As a debugging aid, almost all errors are produced as a result of the one of the following conditions: 1 - Improper sequencing of function calls. 2 - Incompatible input data file and data product header include file (e.g., user included "AVHRR.h" but is trying to read a TM dataset). 3 - Bad data include f'de; perhaps an existing .h file was cloned and improperly altered for a new data product. Errors encountered by the current implementation of these functions will cause an immediate abort (with an error message to the UNIX stderr). This behavior will be altered in a future release to use the ECS error trapping and reporting functions as required in a production environment. The error messages documented here will also be passed to the MODIS log on the MODIS team leader's computer facility (TLCF). The source code contains debug statements that can be 'turned on' at compile time for additional information. These messages are not documented in this User's Guide. 17 MOD_IO_allocateOutputB uffer SYNOPSIS: DESCRIPTION: err = short MOD_IO_allocateOutputBuffer( short , void * ) This function requests that the library obtain memory for an output data buffer from the operating system. The space required is known from the internal data structure associated with the user supplied and the master scan cube. INPUTS: OUTPUTS: short : The output f'de designator returned by the : The address of the user declared instance of the scan cube data structure, into which the library places the updated pointers to the new data. RETURN: err : The return status value; 0 if all went well; error message and program termination if not. EXAMPLE: #include "MOD_IO_prototypes.h" #include "AVHRR.h" AVHRRscancube outputBuffer; COMMENTS: fileout = MOD_IO_openOutputDataset( ... if( MOD IO allocateOutputBuffer( fileout, &outputBuffer ) != 0 ) printf( "allocate error: %dha", err ); When an output buffer is allocated by the library, it inherits the scan cube number of frames from the corresponding "Master Input Scan Cube". ERROR MESSAGES: "No output stream to write to!" "allocateOutputBuffer: Invalid output stream id" "Sorry, output buffer for id = has not been flushed .... can't recover" "Sorry, cannot match next output id with master input buffers" "Output id =, input list = " "can't recover" "Sorry,unable to allocate output buffer for band ""can't recover" 18 MOD_IO_closeDatasets SYNOPSIS: err = short MOD_IO_closeDatasets( void ) DESCRIPTION: This function closes all currently open files. INPUTS: (n/a) OUTPUTS: (n/a) RETURN: err : The return status value is 0 if successful; error messages otherwise. EXAMPLE: #include "MOD IO prototypes.h" #include "AVHRR.h" if( MOD_IO_closeDatasets 0 != 0 ) printf( "Not being able to close files is bad news\n" ); COMMENTS: This function just calls the MOD_IQcloselnputDataset and MOD IO closeOutputDataset functions. See those descriptions for details. ERROR MESSAGES: See MOD_IO_closeInputDataset and MOD_IO_closeOutputDataset function calls. 19 MOD_IO_closeInputDataset SYNOPSIS: err = short MOD_IO_closelnputDataset( short ) DESCRIPTION: This function closes the user specified input dataset file and deallocates (frees) all memory (malloc'ed) associated with the f'de designator . INPUTS: short • The file designator provided to the user by the MOD_IO_openlnputDataset function. OUTPUTS: (n/a) RETURN: EXAMPLE: err : The return status value; 0 for normal operation. #include "MOD_IO_prototypes.h" #include "AVHRR.h" s:: COMMENTS: ERROR MESSAGES: fileln = MOD_IO_openMasterinputDataset( ... if( MOD IO_closelnputDataset ( f'deln ) != 0 ) printf( "Not being able to close files is bad newskn" ); "closelnputDataset: Invalid input stream id" Z 20 MOD_IO_closeOutputDataset SYNOPSIS: err = short MOD IO closeOutputDataset( short ) DESCRIPTION: This function writes the trailer record to the output dataset f'de and deallocates (frees) all memory (malloc'ed) associated with the file designator . INPUTS: short : The file designator provided to the user by the MOD_IO_openOutputDataset function. OUTPUTS: (n/a) RETURN: err" The return status value; 0 for normal operation. EXAMPLE: #include "MOD_IO_prototypes.h" #include "AVHRR.h" fileOut = MOD_IO_openOutputDataset( ... if( MOD_IQcloseOutputDataset ( fileOut ) != 0 ) printf( "Not being able to close f'des is bad news_n" ); COMMENTS: ERROR MESSAGES: "closeOutputDataset: Invalid output stream id" "can't write EOF" 21 MOD_IO_openInputDataset SYNOPSIS: infd = short MOD_IOopenlnputDataset( char * , DataDescriptor * ) DESCRIPTION: This function opens a dataset in read only mode. The file name is supplied by the user in the character string . The dataset header is read from the disk file and compared with the contents of the data descriptor passed to this routine, INPUTS: char * : The user specified fully qualified input dataset file name. DataDescriptor * : The address of the formal name of the data descriptor as defined in the associated data product header include file. OUTPUTS: (n/a) RETURN: infd : The positive data control block (dcb) number that the library uses in subsequent functions to designate which input stream is being accessed. EXAMPLE: #include "MOD_IO_prototypes.h" #include "AVHRR.h" #define FILENAME "/user/data/modis/Level. 1B/mydata.test.data.sim" inputFileDcb = MOD_IO_openlnputDataset( FILENAME, &AVHRRheader ); COMMENTS: The formal name "AVHRRheader" must exist in the data product header include file "AVHRR.h". This applies to all data product header include files. The user must not alter the returned value: "inputFileDcb" in the above example. An error condition will be reported if a master input dataset is not already open. The variable "infd" can be an array element, e.g., "infd[2]". (Error messages on next page) 22 ERROR MESSAGES: "Sorry,youmustspecifyamasterscancubefile first" "Sorry,youhavealreadyusedupall inputDCBs;max= .... can'trecover" "Sorry,unableto openinputfile " "can'trecover" "Can'treadscancubevolumeheader" "Sorry,wronginstrument!" "Sorry,wrongdomain!" "Sorry,wrongdatatype!" "Sorry,noVersion=!" "compiledversion:,datasetversion" "Incompatibilitybetween library versionanddataset" "Sorry,nonbands=!" "compiled# bands!=actualdatastreamheader# bands" "Sorry,nonlines=!" "compiled# lines [=actualdatastreamheader# lines" 23 MOD_IO_openMasterInputDataset SYNOPSIS: infd = short MOD_IO_openMasterlnputDataset( char * , DataDescriptor • ) DESCRIPTION: This function opens a master dataset in read only mode. The file name is supplied by the user in the character string . The dataset header is read from the disk file and compared with the contents of the data descriptor passed to this routine. INPUTS: char * • The user specified fully qualified input dataset t'de name. DataDescriptor * • The address of the formal name of the data descriptor as defined in the associated data product header include file. OUTPUTS: (n/a) RETURN: infd • The positive dcb number that the library uses in subsequent functions to designate which input stream is being used. EXAMPLE: #include "MOD_IO_prototypes.h" #include "AVHRR.h" inputF'tleDcb = MOD_IO__openMasterlnputDataset( FILENAME, &AVHRRte.a_r ); COMMENTS: This routine calls MOD_IO_openlnputDataset internally to do most of the work. ERROR MESSAGES: See MOD_IO_openlnputDataset reference section, plus: "Sorry, you cannot open more than one Master input file" "ndcb_in = " "Can't recover" 24 MOD_IO_openOutputDataset SYNOPSIS: outfd = short MOD_IO_openOutputDataset( char * , DataDescriptor * ) DESCRIPTION: This function opens a dataset in write mode. The file name is supplied by the user in the character string . The type of the data product is specified by the user as a pointer to the data descriptor "" INPUTS: char * : The user specified fully qualified output dataset file name. DataDescriptor * : The address of the formal name of the data descriptor as defined in the associated data product header include file. OUTPUTS: (n/a) RETURN: outfd : The positive dcb number that the library uses in subsequent functions to designate which output dataset is being used; -1 for no more library internal dcbs available. EXAMPLE: #include "MOD IO_prototypes.h" #include "AVHRR.h" outputFileDcb = MOD_IO_openOutputDataset( FILENAME, &AVHRRheader ); COMMENTS: The formal name "AVHRRheader" must exist in the data product header include file "AVHRR.h". This applies to all data product header include files. The user must not alter the returned value: "outputFileDcb" in the above example. The variable "outfd" can be an array element e.g., "outfd[ 1]". ERROR MESSAGES "Sorry, you have already used up all DCBs; max = " "Sorry, you must specify a master scancube file fast" "Unable to open output f'de " "can't recover" "can't support writing multiple scancube types within a scancube now!" "Can't write output header" 25 MOD_IO_printDataDescriptor SYNOPSIS: void MOD_IO_printDataDescriptor( DataDescriptor * ) DESCRIPTION: The function prints the contents of the user specified data descriptor "" to stderr. It is useful for informational purposes when debugging user written algorithms and data product header include f'des. INPUTS: OUTPUTS: DataDescriptor * : The address of the formal name of the data descriptor as defined in the associated data product header include file. (n/a) RETURN: (n/a) EXAMPLE: #include "MOD IO_prototypes.h" #include "AVHRR.h" fdin = MOD_IO_openMasterlnputDataset( "myfile", &AVHRRheader ); MOD IO printDataDescriptor( &AVHRRheader ); COMMENTS: The data descriptor contents are printed to stderr in labeled and tabulated form. INFORMATION MESSAGES: "Read Data Descriptor:" name = " # data types/scancube = " # bands = '* (bands are equivalent to parameters) # lines = " " resolution for band = " (multiple occurances) " size of band = bytes" (multiple occurances) ERROR MESSAGES: (none) 26 MOD_IO_readSwath SYNOPSIS: index = long MOD_IO_readSwath( short , void * ) DESCRIPTION: This function reads a scan cube from the user specified input dataset. INPUTS: short : The input file designator returned by the MOD_IO_openMasterlnputDataset or MOD_IO_openlnputDataset routines. OUTPUTS: void * : The address of the user defined scan cube data structure, into which the library places the updated pointers to the new data. RETURN: index->if positive: a sequential index number of the scan cube that has just been read, starting from one (1). index->if negative: an error return status value: - 1 for an end of file (EOF) EXAMPLE: #include "MOD IO .prototypes.h" #include "AVHRR.h" AVHRRscancube ibuf; inputDcb = MOD_IO_openMasterlnput( ... index = MOD_IO_readSwath( inputDcb, &ibuf); COMMENTS: This call brings in the next scan cube from the specified dataset contained on disk into library managed and allocated memory. Pointers to the scan cube elements are set in the user's data structure to point to the data in the library data area. The user may access any band (or parameter) and pixel through the scan cube data structure definitions. The data structure also provides access to the spatial sizes of all bands (or parameters). See code examples elsewhere in this document. (Error messages on next page) 27 ERROR MESSAGES: "No input streamto read!" "readSwath: Invalid input stream id" "Can't read scancube header" :: "Sorry, cannot overwrite scancube with index=" "Warning: scancube header doesn't have an id" "scancube header doesn't have # frames" "Bad science data read; can't recover" "Bad SD cal data read; can't recover" '!Warning: scancube header doesn't have SD mode" "SD present, but incompatible .h file" "Bad SRCA cal data read; can't recover" "Warning: scancube header doesn't have SRCA mode" "SRCA present, but incompatible .h file" "Bad BB cal data read; can't recover" "BB present, but incompatible .h file" "Bad SV cal data read; can't recover" "SV present, but incompatible .h file" "Bad EM cal data read; can't recover" "EM present, but incompatible .h file" "Sorry, can't allocate space for bufindex , band " "Sorry, incomplete read; wanted , got " "Sorry, can't allocate space bytes data, bufindex " 28 MOD_IO_writeSwath SYNOPSIS: err = short MOD IO writeSwath( short ) DESCRIPTION: This function writes the contents of the current output buffer to the dataset specified by the user supplied outputFileld, where outputFileld was obtained from the MOD__IO_openOutputDataset call. INPUTS: short : The value returned by the MOD_IO_openOutputDataset call. OUTPUTS: (n/a) RETURN: err" The return status value: 0 if every thing is OK, -1 if there is no output file to write tO. EXAMPLE: #include "MOD_IO_prototypes.h" #include "AVHRR.h" OutputFileld = MOD_IO_openOutputDataset( ..... err = MOD_.IO_allocateOutputBuffer( .... (place values in the output buffer) err = MOD_IO_writeSwath( OutputFileld ) COMMENTS: ERROR MESSAGES: "writeSwath: Invalid output stream id" "Sorry, no output buffer has been allocated for product # (nothing to write!) .... can't recover" "Sorry, unable to write bytes; only wrote bytes for output data stream #" "can't recover" "ERROR: cannot match next output id with master input buffers" "Output id =, input list = " "can't recover" "arglist write too long" (internal error) "Can't write output scancube header" 29 LIBRARY ACCESS AND INSTALLATION The library source code, make f'des, and test programs are contained a UNIX tar fide. This may be obtained from the/pub directory on modis-xl.gsfc.nasa.gov via anonymous ftp and contains a README file with the latest details and dataset contents descriptions. It also contains an INSTALL file with full installation instructions. The following list shows a sample of the tides contained in the tar distribution: Table 2. List of Files README INSTALL MOD_IO_lib.c MOD_IO_prototypes.h MOD_IO_lib.h MOD_IO_headerDefs.h DataDescriptor.h AVHRR.h (additional files) a file containing the latest information with a description of all the other files in the distribution installation instructions for adding this library to your suite of computer based tools the MODIS SDST utility library (all library functions) ANSI function prototypes for library functions data structures for scan cube buffer descriptions and DCBs; NOT explicitly needed by application programs defines for constants in the library specific to the dataset headers (also used by non MODIS scan cube generators) data structure definition for formal Data Products a sample AVHRR data product header file sample algorithms and test cases (see the README file for details) 30 APPENDICES Appendix A - MODIS Spatial Domain Descriptions The MODIS Scan Cube Spatial Domain The MODIS instrument generates raw (digital counts) data for each detector within the instrument. This set of detectors scans across (perpendicular to) the satellite orbit ground track to produce an amount of data designated as a scan cube (see Figure 7). Multiple scan cubes are collected together to form a data granule. Multiple granules are collected into an orbit. The sizes of this basic scan cube are summarized below. Scan cube details are found in the MODIS Data Structure, Rates, and Volumes document as referenced in the Bibliography. Trailer record veflaUeler_ Engr/MemDump _/ 2 equivalent frames I" / OBC Frames _" / I 4 sots of 15 to 50 frames I 1 Science Frames commendable, 1354 ourmn_ / Header record-----_ variable length 500m bands 250m bands < 10 to 40 scan lines Figure 7. MODIS Level-1 simplified scan cube. 31 The MODIS instrument has the capability of setting the across track size of the scan cube by ground command. The current information indicates that this is not expected to be varied often, but plans are being considered to lower the data transmission rate possibly by varying the across track size as a function of latitude. In any case, the algorithm developer should be aware that the number of across track pixels is not a constant. For the scan cube domain, the algorithm developer deals with a single band of MODIS data coveting an entire scan cube swath. The sizes given are in pixels, not kilometers. The pixel ground coverage sizes vary with the scan angle from nadir. The collection of detector values from all the bands that cover a single nominal (at nadir) one (1) kilometer area are called a spatial element. All bands of data are specified to be co-registered within 20 percent of the spatial element IFOV and are expected to be co-registered within 10 percent of the IFOV. Note that this specification applied to the registration knowledge. A table of band to band offsets will be published by the MODIS SDST and may be included as part of the MODIS geolocation. Table 3. MODIS Level-lB Data Product Scan Cube Sizes ITEM HOW MANY FIXED OR VARIABLE number of bands 36 fixed across track pixels; bands 1 and 2 5,416 variable across track pixels; bands 3 to 7 2,708 variable across track pixels; bands 8 to 36 1,354 variable along track pixels; bands 1 and 2 40 fixed along track pixels; bands 3 to 7 20 fixed along track pixels; bands 8 to 36 10 fixed geolocation data 8 fixed The MODIS Level-1A Data Product consists of the raw counts with a separate Geolocation Data Product associated with each individual spatial element. The raw counts are available for the Earth view as well as the various onboard calibrator (OBC) views. This raw data product is expected to be of use only to the Level- IB calibration process. (The OBC access functions are described in an appendix.) The science algorithms are expected to use the at-satellite radiances contained within the MODIS Level-lB Data Product. This data product contains the radiance energy generated by viewing the Earth and measured at each detector within the MODIS scanning swath. These values are spatially coincident with the raw counts and have not been spatially resampled. The geolocation information associated with the MODIS Level-1A product therefore applies to, and will be associated with the Level-lB Data Product. The Level-1A dataset components will be synchronized with the Level-1A Metadata and the associated scan cube domain Geolocation Data Product. The Level-1B dataset components will similarly be transparently aligned with the Level-1B Metadata and this same scan cube domain Geolocation Data Product. All of the data sets are accessible from the identical library calls, via separate data structures. 32 TheMODISLevel-1ADataProductalsocontainsinformationderivedfrom theonboardcalibrators (OBCs)in two forms.Thefirst dataform isalsoin thebasicstructureof ascancube,but is measuredby imagingtheOBCradiancesourcesin placeof theEarthbasedradiances.Therewill alsobeoccasional Moon viewingdatain eitherorbothof theEarthviewsandOBC spaceviews.Thesecondform of OBC datais derivedfrom theengineeringparametermeasurementshataremultiplexedwithin thepacketdata. Theseareseparatedinto aseparatecalibrationdataproductfor useby theLevel-1Bprocess.Thelibrary callsto accesstheseportionsof theMODISdataarenot includedin thisdocument. TheMODIS Level-2DataProductdatasetscanalsobeaccessedandgeneratedwith library callsincluded in thisSDSTutility library.ThisallowspreviouslygeneratedMODISLevel-2productsto beusedasinputs for thecreationof otherLevel-2DataProducts. DataProductsin thisscancubedomainwill notbetemporallyor spatiallycoincidentwith anyotherscan cubesor domainsandmustberesampledor otherwiseaggregatedintoarectifiedor mappeddomainfor truespatialregistration.All truespatialoperationsin thescancubedomainareleft to theuser. The MODIS Rectilinear Domain Algorithms that operate on a single spatial pixel area at a time, and do not depend on the exact direction and distance to adjacent spatial pixels, can generate science values in the scan cube geometry domain. These values can then be mapped directly from the scan cube domain onto a Level-3 map projection or similar geobased database. However, algorithms that depend on spatial directions and distances can easily use either a rectilinear MODIS data domain or a mapped data domain as input when deriving science Data Products. The rectilinear domain is defined to be a coordinate system that uses the spacecraft ground track as one axis, the scan perpendicular to the ground track as the second axis, and radiance data values resampled to a uniform distance along each of these axes.. Only the MODIS Level-1B radiance values and MODIS Level-2 Data Products are appropriate for the rectilinear domain. The MODIS Level-lA data is best represented in the scan cube domain and is not appropriate to the rectilinear domain. Note that MODIS data that uses or passes through a rectilinear domain will be resampled twice: once from scan cube to rectilinear, and again going from rectilinear to mapped. For this reason, it is best to keep algorithms in the scan cube or mapped domain. The MODIS Mapped Domain This designation is used for all Data Products that have been transformed, via a map projection, onto a fiat surface such as a piece of paper or display pixel indices (the electronic equivalent). A map projection can be defined as the orderly transfer of Earth surface positions to corresponding points on a fiat surface. This definition can be further generalized into a three dimensional hierarchical data structure such as a quad tree, where each area on the flat surface is successively divided into smaller areas. Since the surface of a sphere 33 cannotbe forceddirectlyontoa fiat plane without simplifying approximations, several types of maps are expected to be used to represent either the entire Earth global data sets or sections of the globe such as continents. Mapped domain library calls will be def'med in a separate document on MODIS Level-3 toolkit functions. 34 Appendix B - Data Product header include file description A series of data product header include f'des that describe the data structures for: the input and output formal Data Products, static arrays of characters describing the dataset contents, ANSI function prototypes, and global variables are included as part of the library distribution. These are for use with "C" programs only. FORTRAN equivalents are not available at the present time. Note that FORTRAN90 will enable the use of these types of headers while ANSI FORTRAN77 will not. Each formal Data Product has an accompanying header file that contains the product specific information which contains dynamic variables and is accessible to the science algorithm developers. The set of all of these Data Product header files will be created in cooperation with the algorithm writers, maintained by the MODIS SDST A'I'T members, and made available for use by all algorithm developers. These headers will be coordinated with the science data product office (SDPO) definitions for the formal Data Products and parameters and placed under configuration management by the SDST. Each Data Product header include file contains the data structure definitions for each of the parameters that are a part of every formal Data Product. Data product parameters are the components of each formal Data Product such as the data product itself, values for the precision of the data product, all quality assurance (QA) parameters, and intermediate informal data products associated with the formal Data Product. The data types (short, long, float, etc.) for each parameter are defined here along with the sizes, resolutions, and dimensions of the spatial extent of each parameter. Two "C" code modules are also included; one to initialize the data structure (similar to a constructor in C++) and one to set the pointers appropriately for user access. These are "hidden" in this file so that the user need only alter the algorithm source code to include the appropriate file without the concern of linking separate code modules. There could be situations where it is desirable to put these code modules into a separate file. If so, these situations can be easily dealt with on an individual basis. Note that the individual bands of MODIS instrument data in the Level-1A and Level-1B Data Products are equivalent (for library access) to data product parameters and are accessed by the user via the same library call mechanism. The include files required for algorithm development are: "MOD_rO__Drototypes. h", " ", and "". ANSI prototypes for all library calls are included in the "MOD tO prototypes, h" file. The Data Product detailed descriptions are included in a header file that is unique to each instance of a Data Product, either input or output (" <-rnpul: Product. h>" and/or "" ). A fully commented sample data product header file for AVHRR data in a MODIS scan cube format is included in Appendix D. The "DataDescrS.ptor. h" include file, referenced by the <*. h> files, defines a flexible data format which is then used by the product specific < *. h> files to further describe each particular formal Data Product. Each product specific header file also includes two code modules which are invoked by the library. The first module, an initialization section, is called when each input dataset is first opened. This module customizes the DataDescriptor data structure, which is then compared by the library with information in the actual dataset volume header. The second module is called each time a scan cube is read into memory, or an output scan cube is allocated. This code defines and passes the correct data pointers to the user's program for accessing the input or output scan cubes that are actually in the library memory space. 35 FORTRAN(FTN77)accessiblefunctioncallsmaybewrittenata laterdateif thereis auserdemandfor FTN77library functions.If written, theseareexpectedto be"wrappers"to the"C" functioncalls thatallow FTN77accessto datastructurecontents.Theywill allow algorithmswrittenaccordingto F"rN77standards to haveaccessto theMODIS data,butnotefficiently.Exceptionsto theFTN77ANSI standard(suchas includefiles)will haveto be takento accomplishthis task.Notethatalgorithmswritten in FORTRAN90 will allowdirectaccessto datastructuresviapointermechanismsthataresimilar to the"C" language.This is theSDSTrecommendedapproachafterapprovalfor algorithmdevelopmentin FORTRAN. 36 Appendix C - The DataDescriptors.h Library Include File Description This is the include file used by the library and referenced in the formal Data Product header include files. It declares the data structure for the data description area which contains pointers to data product unique substrucure definitions. /* */ /* MODIS scancube i/o library */ /* SDST */ /* Virginia Kalb, NASA/GSFC 920.2 */ /* Thomas Goff, RDC */ /* */ /* RCS keywords: *I /* SHeader: DataDescriptor.h,v 2.1 94/08/16 09:57:49 gk Exp $ */ /* SDate: 94/08/05 09:57:49 $ */ /* $Source: /disk5/modis/sim/MOD_IO/RCS/DataDescriptor.h,v $ */ /* */ /* */ #ifndef DATADESCRIPTOR #define DATADESCRIPTOR struct FPTR { void (*fptr)(); This declares a function pointer data type to facilitate the subsequent declaration of an array of function pointers needed for the Level-1B data. The MODIS Level-1B subcubes contain the Earth (science) view and the four sets of OBC data (SD, SRCA, BB, Space view). ); typedef struct FPTR FPTR; struct DataDescriptor void (*init)(); short nptrs; FPTR *set; char *name; long nbands; long nlines; ( <- Entry point to the initialization routine. <- How many pointers to subcubes are required. <- Pointer to the array ofsubcubes. <- The dataset formal name, contained within the dataset. <- The number of parameters (bands) in this formal Data Product. The number of lines (detectors) in the along track direction for the finest spatial resolution for this formal Data Product. For example, this corresponds to the number of along track detectors for MODIS bands 1 and 2 which is 40. long *nres; For each parameter (band), this is the resolution divisor to obtain the actual number of along track detectors for that parameter (band). For example, MODIS bands 3 to 7 have an nres equal to 2 (20 detectors in the along track direction), and bands 8 to 36 have an nres value of 4 (10 detectors in the along track direction). 37 long *nsize; The size in bytes of each parameter value. For example, MODIS band 1 has 2 bytes per readout, 40 detectors along track, and 4 times the nominal 1354 samples along scan for a final value of 433280 bytes. ); typedef struct DataDescriptor DataDescriptor; #endif 38 Appendix D - Data Product Header example This section contains an example of a generic Data Product header file with embedded comments. A machine readable copy of this file is available as part of the library distribution. The source code is in the Courier typeface and comments following each section of code are in the normal Times Roman typeface. The formal Data Product header include f'de contents are divided into five (5) main sections. The first section defines header includes and internal definitions. The next two sections set up the data structures for each element of a parameter within a data product, and the order of the parameters within a scan cube. The remaining two sections are functions that populate (provide instance specific contents for) the dataset valuables and each scan cube variable respectively. /* ./ /* MODIS scancube i/o library */ /* SDST ./ /* Virginia Kalb, NASA/GSFC 920.2 */ /* Thomas Goff, RDC */ /* ./ /* RCS keywords: ./ /* $Header: PRODUCT.h,v 2.0 94/08/05 17:01:26 gk Exp $ */ /* $Date: 94/08/05 17:01:26 $ ./ /* $Source: /disk5/modis/sim/MOD_IO/RCS/PRODUCT.h,v $ ./ /* ./ The RCS revision cont_lsystem is usedinternallyforconfigumtion managementduring thecode developmentstage. #ifndef PRODUCT #define PRODUCT The abovecpp preprocessordi_ctivespreventduplica_ compiling ofthis module from occurring. #include #include These are the standard UNIX header includes from the standard operating system location for system include files. They are used for both ANSI prototyping and function macros. #include "DataDescriptor. h" This header file contains the MODIS library internal structure that holds values that are fixed for a given product. It contains the data structure specifications for the PRODUCTheader. data structure. #define PRODUCT HEADER_STRING "Product-specific identifer" This define statement declares the formal name of each user's Data Product. No spaces are allowed. This character string will contain something like: "Level-1B_radiances" for the MODIS Level-1B Data Product. The spatial domain designation is contained within the datasets. This allows differing spatial domains containing the same data types to be used interchangeably by the library and hence the user. 39 #define DUMMY 1 This is a dummy number (could be any number actually) used to satisfy the compiler when declaring the variable dimension component in a multi-dimensional array. The data structures that allow for single or two dimensional access to the data are illustrated later in this listing. typedef double PRODUCTparml; typedef long PRODUCTparm2; typedef unsigned char PRODUCTparm3; These type definitions specify the data type of the formal Data Product parameters. In this case, the Data Product has three Parameters (these are bands for Level-lA and Level-lB). The first parameter is of type double (64-bit floating point / real), the second is of type long (32-bit integer), and the last parameter is of type single byte (8-bit). The data type is user-driven. /* Structure for individual bands/parameters: */ struct PRODUCTPARMI { PRODUC_oarml *data; long nAlongTrack; long nAlongScan; long nPixels; }; typedef struct PRODUCTPARMI PRODUCTPARMI; This data structure declaration defines the first product parameter (the data part of which has been specified above) to be a data structure containing a pointer to the spatial data array for this parameter, an along track and across track dimension, and a total number of pixels. Note that the data dimensions are adjusted by the library to reflect the resolution of each individual parameter. struct PRODUCTPARM2 ( PRODUCqloarm2 *data; long nAlongTrack; long nAlongScan; long nPixels; }; typedef struct PRODUCTPARM2 (Same as the above description.) struct PRODUCTPARM3 ( PRODUCTparm3 *data; long nAlongTrack; long nAlongScan; long nPixels; ); typedef struct PRODUCTPARM3 (Same as the above description.) /* Structure for scancube output: struct PRODUCTscancube { PRODUCTPARMI pl; PRODUCTPARM2 p2 ; PRODUCTPARM3 p3 ; PRODUCTPARM2; PRODUCTPARM3; */ 4O }; typedef struct PRODUCTscancube PRODUCTscancube; /* User should declare an output structure like /* PRODUCTscancube obuf; */ this: */ This statement declares the data structure type for user access as a single linear an-ay. The examples given in the main section of this document in the scan cube domain functions - introduction section, illustrate its usefrom auser'sperspective. /* Structure for 2d scancube output: */ /* CAREFUL! The rightmost array dimension MUST MATCH THE */ /* ACqUAL # OF SPATIAL ELEMENTS for that band! */ /* */ struct PRODUCT2Dparml { PRODUCTparml xy[DUMMY][10]; ); typedef struct PRODUCT2Dparml PRODUCT2Dparml; More type definitions: in this case to Blow two dimensionl access to the spati_ data. The array name xy is used as a pointer to this two dimensional array. Note Mat the requirement Mat the varying dimension, co_esponding to the MODIS variable number of frames across scan cubes, _ in the fist set of bmcke_ N with any number (DUMMY) used as a place holder. The number m Me second bracke_ N must coITespond to the actual along track (across scan) size for each parame_r. The order of Me indices in "C" is reversed _om those in FORTRAN. struct PRODUCT2Dparm2 { PRODUCTparm2 xy[DUMMY][10]; }; typedef struct PRODUCT2Dpaz]n2 PRODUCT2Dparm2; (Same asthe otherparameters) struct PRODUCT2Dparm3 { PRODUCTparm3 xy[DUMMY][10]; ); typedef struct PRODUCT2Dparm3 PRODUCT2Dparm3; (Same asthe otherparame_rs) /* Structure for individual bands/parameters (2D): */ struct PRODUCTPARMI_2D { PRODUCT2Dparml *data; long nAlongTrack; long nAlongScan; long nPixels; ); typedef struct PRODUCTPARMI_2D PRODUCTPARMI_2D; These two dimensional access methods are effectively the same as the previous typedef's for PRODUCTPARM 1: *data is a pointer to the actual scan cube binary data. struct PRODUCTPARM2_2D { 41 PRODUCT2Dparm2 *data; long nAlongTrack; long nAlongScan; long nPixels; ); typedef struct PRODUCTPARM2_2D (Same as PRODUCTPARM I_2D) struct PRODUCTPARM3_2D { PRODUCT2Dparm3 *data; long nAlongTrack; long nAlongScan; long nPixels; ); typedef struct PRODUCTPARM3_2D (Same as PRODUCTPARM I_2D) struct PRODUCT2Dscancube ( PRODUCTPARMI_2D pl; PRODUCTPARM2_2D p2 ; PRODUCTPARM3_2D p3 ; ); PRODUCTPARM2_2D; PRODUCTPARM3_2D; typedef struct PRODUCT2Dscancube PRODUCT2Dscancube; /* User should declare an output structure like this: */ /* PRODUCT2Dscancube obuf; */ This statement declares the data structure type for user access as a two dimensional array. The examples given in the main section of this document in the scan cube domains functions - detailed section, illustrate its use from a user's perspective. Note that ideally, one scan cube definition would be given in each header file. Both the one and two dimensional definitions are given in this "PRODUCT.h" sample header file for illustration purposes. Other definitions could be made at the user's request, but care must be exercised to propagate any changes to the intrinsic function code and other modules within the header file. Now thatthe datastmctu_sand declarationsa_ over with, we canpopula_ instancesofthesecomplex variables with the valuesappropfiate to this _rmal Data Productbyinvoking actualexecu_blecode. void PRODUCTinit(); void PRODUCTsetptrs(void **, PRODUCTscancube *, long); Static DataDescriptor PRODUCTheader = {PRODUCTinit,PRODUCTsetptrs,NULL, 0,0,NULL,NULL); 42 Le_ start with the ANSIpromtypesfor the functions that follow, and mitialize(declam space for) the dam descfiptorstmctu_.Thisisexecu_d _reach datasetopen. void PRODUCTinit() { static short executed = 0; if(executed > 0) return; /* don't want or need to execute this more than once!*/ executed = i; The variableexecutedisusedtoinsum _atthis _ncfionisexecumd only once. PRODUCTheader.name = PRODUCT_HEADER_STRING; PRODUCTheader.nbands = 3; PRODUCTheader.nlines = 40; PRODUCTheader.nres = (long *)malloc(PRODUCTheader.nbands*sizeof(long)); PRODUCTheader.nres[0] = 4; PRODUCTheader.nres[l] = 4; PRODUCTheader.nres[2] = 4; PRODUCTheader.nsize = (long *)malloc(PRODUCTheader.nbands*sizeof(long)); PRODUCTheader.nsize[0] = sizeof(PRODUCTparml); PRODUCTheader.nsize[l] = sizeof(PRODUCTparm2); PRODUCTheader.nsize[2] = sizeof(PRODUCTparm3); } This function populates the PRODUCTheader. structure with the values appropriate to this formal Data Product. These values apply to the entire dataset. String sizes are determined at compile time, and nres and nsize arrays are allocated at run time. See the "DataDescriptors.h" section in the appendix to this document for the structure declaration which def'mes the data types for each structure element name. void PRODUCTsetptrs(void **obuf, PRODUCTscancube *odata, long nframes) This function is entered on each allocation of a scan cube. It users the library supplied pointer odat a, and library supplied pointer *obuf to the user declared buffer obuf, along with the library supplied nframes to calculate the values of the scan cube dimensions nAlongTrack and nAlongScan and the scan cube size nPixels. The value of nframes is extracted by the library from each scan cube header. These values, along with the starting addresses (pointers) of the data array, are then placed into the user's instance of the formal Data Product data structure. This is repeated for each parameter separately, thereby allowing the spatial dimensions to vary across bands (parameters). { long across, down; odata->pl.data = obuf[0]; down = PRODUCTheader.nlines/PRODUCTheader.nres[0]; odata->pl.nAlongTrack = down; across = nframes/PRODUCTheader.nres[0]; odata->pl.nAlongScan = across; odata->pl.nPixels = down*across; odata->p2.data = obuf[l]; down = PRODUCTheader.nlines/PRODUCTheader.nres[l]; 43 odata->p2.nAlongTrack = down; across = nframes/PRODUCTheader.nres[l]; odata->p2.nAlongScan = across; odata->p2.nPixels = down*across; odata->p3.data = obuf[2]; down = PRODUCTheader.nlines/PRODUCTheader.nres[2]; odata->p3.nAlongTrack = down; across = nframes/PRODUCTheader.nres[2]; odata->p3.nAlongScan = across; odata->p3.nPixels = down*across; return; } #endi f End of this data product header include file. 44 Appendix E - MODIS Dataset Internal Format This section details the contents of the dataset that are ingested and created by the SDST utility library. The dataset contents implement a free field technique to determine the sizes and location within the Dataset for the various components of the Dataset. This allows data from multiple instrument sources and multiple input and output Data Products to be handled by the library in a manner that is transparent to the user. VOLUME HEADER SCANC_E HEADER SCAN CUBE CONTENTS SCAN CUBE HEADER SCAN CUBE CONTENTS SCAN CUBE HEADER SCAN CUBE CONTENTS (etc.) TRAILER Figure 8. Overall dataset contents. Each Dataset is composed of three main sections: a Dataset volume header, multiple instances of the scan cube header and data, and a trailer as shown in Figure 8. All headers sections and the trailer section contain information in ASCII form that can be read (carefully) with any file browser or editor. Recommended file browsers are the UNIX od command, the gnu less program, and the SDST written fd program. These programs can examine mixed ASCII and binary files while not creating problems with communications protocols. The od and fd programs can handle byte offsets for binary data that is not aligned on word boundaries. The UNIX head, tail, and more programs will cause problems with terminal based communications when examining binary data. The Dataset is organized as a self describing data structure. Information is contained in each section that informs the library of the sizes of the following sections. This technique allows a byte stream oriented operating system (i.e. UNIX) to define variable length and mixed type (ASCII / text and binary) records. Data is organized in this byte stream by placing the along track pixels in adjacent words of memory to form a vector, vectors are then sequenced across track to form a band or parameter spatial area, then multiple areas are sequenced to form the entire scan cube. This ordering in memory allows the across track 45 dimensionto varyamongscan cubes without prohibiting two dimensional access by "C" and FORTRAN compilers. The illustration in Figure 9 assumes a data product composed of three parameters, each of type byte (unsigned character or Integer*l) with nominal spatial resolutions of 250 meters, 500 meters, and 1.5 kilometer. This is a fictitious instrument for illustration purposes only. The reader is encouraged to examine the small datasets included in the library distribution for actual parameter values and compare these with the sizes in the diagram and described in this text. 1 1 71_/ 4 5 6 289292 295 290293296 291 294 369 299 3OO 306 37O 371 Across Track 1111 III! IIII III ..... 1 I I I 11 11 II 1 I I r-- ==- 375 376 nres= l 288 elements nres=2 72 elements nres=6 8 elements (numbers in the boxes are the array elements in a byte stream) Figure 9. Memory allocation for variable coincident spatial sizes. The numbers in this diagram represent the linear addresses of the byte stream. The three spatial areas with resolutions of 1, 2, and 6 can be spatially registered on the ground, but are sequential in computer memory. The illustration given here does not represent any of the current datasets and is used for illustration purposes only. The Dataset volume header contains information that is required by the library to recognize and interpret the contents of the Dataset. This includes the source of the data (must be "MODIS"), the type of data (unique to each formal Data Product), and the version of the Dataset contents. This Dataset version must agree with, or otherwise be compatible with, the library. It is the library's responsibility to verify this. Other items in this header are compared with the DataDescriptors in the library as a sanity check to ensure that the library and user code are using the same data formats and types. The remainder of the volume header contains additional items unique to each Data Product and any comment fields included by the Dataset generator. The scan cube section contains a scan cube header and data in a binary form. The scan cube header is also in the same free field format as the volume header. Dataset parameter values are reset as each scan cube is accessed. This allows the handling of binary data that can change with each scan cube. 46 Thebinarysectioncontainstheactualdatavaluesin the data type defined within each Data Product "" include file. The data is treated by the library as a byte stream whose size is the across track size times the along track times the number of bytes in the data product, where the number of bytes in the data product is the sum of the number of bytes for each parameter. The trailer contains the key character string "EOF" to designate a logical end of f.de. All other tokens in the trailer are ignored in this release, but may contain metadata information in the future. All of the headers contain ASCII data in a free field format. Each field, called a token, consists of all the characters that are delimited by 'white space', where white space is defined to be blank (space) characters, tab characters, or commas. The fast three tokens are recognized by the library as character strings that are compared with information in the "" headers associated with each format Data Product. All tokens that contain an equal (=) sign are recognized as "C" type variables and values. These variable names are formal names used by the library to determine the sizes of Data Product parameters, spatial resolutions of the pixel elements, and the resulting sizes of the binary data that follow. All other tokens in the header are ignored by the library and can contain any user defined comments. Details are given in the following table. Table 4. Header Parameter Names FORMAL NAME DESCRIPTION UNIQUE TO: Version The version number of the library that can access this Dataset dataset headerLines The number of lines contained in this header section (terminated dataset by \n) Number of bands or parameters in the data structure Number of along track lines of data within each scan cube at resolution 1 Multiple of the smallest instrument IFOV (for MODIS this is 250 meters * nres) Number of bytes per data element number of elements across track at resolution 1 for the Earth view unique ID within this data set for each number of elements for the Solar Diffuser view current SD mode: "DCrestore I Solar I Screen" number of elements for the SpectroRadimetric Control Assembly "radiometric I spectral I spatialAlongScan I spatialCrossScan" number of black body viewing frames the BB temperature (an array in the future) nbands dataset nlines dataset nres dataset nsize dataset nframes scan cube scancubeld scan cube nSDframes scan cube SDmode scan cube nSRCAframes scan cube SRCAmode scan cube nBBframes scan cube BBtemp asynchronous 47 nSVframes numberof spaceview elements scancube DATASET contents example Here is an abbreviated example output from the UNIX less program of a test scan cube dataset. Line numbers have been added. Excess binary data has been deleted for illustration purposes and the dataset contents are incomplete. The reader is encouraged to look into the contents of the datasets provided in the distribution for real life examples. contents dzscription 1 - MODIS Scancubes AVHRR_albedo Version=l.0 headerLines=3 2 - nbands=5, nlines=40, nres=4 nsize=2 3 - This is a comment line 4- nframes=64 scancubeld=l 5 - ^@ANA@AOA@AOA@AOA@ASA@^T^@^V^@^V 6 -^AATAA^TAA^TAAATAA^TAA^UAA^T^AAT 7 - nframes=64 scancubeld=2 8- AA[m#[5mAA[m#[5mAA[m&[5mAA[m&[5mAA[m%[5mAm^A[m&[5mAA[m%[5mAA[m%[5mAA[m 9 - ^A^UAAAT^A^T^AATAA^TAAAUAAAUAAAUAAAUAAAUUAAAUAAAUAAAU^A^UAAAU[m 10 - nframes=64 scancubeld=3 11 - A@ATA@AW^@^WA@AW^@^V^@^UA@^UA@^U^@ @^W^@^UA@^UA@^UA@^UA@AT[m 12 - ^A^SAAAUAAAT^A^T^AASAAASAA^SAA^SAA^SAANW^A^T^A^T^A^SAAt_F_A^SAAAS[m 13 - nframes=64 scancubeld=4 14 - ^@^T^@^R^@ARA@^SA@^U^@^V^@^V^@^V^UA@^V^@^V^@^V^@AV^@AW^@^W[m 15 - ^A^SAA^SAAASAA^SAA^SAA^SAA^SAA^T^A^T^AASAA^SAA^SAA^SAA^T^A^T^A^T[m 16 - nframes=64 scancubeld=5 17 - ^@AUA@^UA@ATA@ATA@AT^@AUA@AUA@^TA@^TA@ATA@^UA@^UA@AV^@AV[m 18 -^AASAA^SAAASAA^TAAATAAATAAASAAATAAATAA^TAAAT^AASAAASAAASAA^SAAAT[m 19 - EOF Figure 10. Dataset contents example. 48 Thevolumeheaderconsistsof thefirst threelines.Thekey token"MODIS" is requiredto authenticate the dataset as a MODIS format dataset. The token "Scancubes" indicates the spatial domain of the dataset which the library uses to determine which of the "=" tokens are required. The next token "AVHRR_albedo" indicates the formal Data Product name and must agree with the corresponding string in the "" file for this Data Product. The "Vers ion= 1.0" token validates the library version with the dataset version. The "headerLines=3" token informs the library that three text lines, terminated by a new line character, define the volume header. The fourth line contains the two required tokens for a scan cube domain dataset. The total number of pixels for a resolution 1 spatial area for the highest resolution parameter, is given in "n f tame s = 6 4". This is divided by the number of lines per scan cube "nl ines=4 0" at each spatial resolution to give the number of across track elements (frames in the MODIS literature). The "s c ancube I d= 1" tells the library the unique scan cube identifier. This number is written by the library into the corresponding output data product scan cube. Lines 5 and 6 in the above example are the ASCII equivalents, in printable form, of each byte of binary data. This has been shortened for illustration purposes. Multiple scan cubes are shown in lines 7 - 9, 10-12, 13-15, and 16-18. The binary data is not terminated by a new line as shown. The next scan cube header begins directly after the binary data and can be anywhere on a displayed line. It is shown left justified for illustration purposes. The final line, 19 in this minimal example, contains the token "EOF". This token is optional but does indicate that the dataset has been terminated gracefully and not truncated abnormally. Each header section can optionally contain the token "headerLines =". If it is not present, a value of one (1) is assumed. If, however, this token is missing and there is more than one header line in the dataset, then the data stream will be hopelessly out of sync since the ASCII data will be interpreted as part of the binary science data. The library will eventually catch this mistake in a subsequent header read and return an error code to the user, but not at the expected processing step. 49 Appendix F - Library Dataset Interactions This appendix presents an overall view of the internal workings of the library. It is not a full library documentation, but is meant to provide a feel for what is happening behind the scenes. The figures in this appendix help the reader understand the relationship between the formal Data Product header include files and the library's use of information in both the data product header include files and the dataset contents. The diagrams in Figure 1! and Figure 12 illustrate the relationship between the input and output data sets, the utility library, and the user code. Data can be transferred, under user control, by the library from the input scan cubes to the output scan cubes without passing through the memory space of the user code. The first figure also illustrates the use of multiple input and output scan cubes within the library space and managed by the library internal DCB structures that incorporate dynamic memory allocation and release. Input Dataset volume header .._ scan cube header" _. scan cube data_ n times trailer Output Dataset volume header scan cube header _ scancube data j¢ n limes trailer Source_Header.h Library Code right instrument? right version of software?/ right data source? 4_ .J right number of bands? _ right number of lines? '= right data resolution? _ number of frames & ID input data _tructure P i Input data structure ! I ,. outputdata structure output data structure / source name .-_ number of bands .-_...f number of lines I../ spatNI resolution Product_Header.h / product name / / number of parameters .// _ _Puati_le;"c_l Inute_n LX _nte_X,User Code pointe% adjustedbyre_iution_-_ factor for each parameter _j ,XX{ pointerto data V3/ Figure 11. Data flow illustration. 50 The second figure illustrates dataset access by surrounding the user's code with the library functions. Dataset contents are accessed via the data structures defined in the data product header include files. Note that non-MODIS data is accessed via include files or other data structures that are not defined by this library. (probagate sizes) _ (check _._ __ \ / (check pro.du,cth.eader\, productheader / " • key include file _ include file _ (check key words)_f _ _ _ words) MODIS dataproduct header include file OU_Ou Science Algorithm \ file \ Auxiliary Data reformattin resampling Auxiliary Dataset Figure 12. Algorithm dataset interactions. 51 (Thispage intentionally left blank) 52 GLOSSARY This section includes the definitions of selected key words used in this document. aggregated algorithm ancillary ANSI A'IT auxiliary AVHRR band Black Body BB View C CZCS DAAC Any of several methods for deriving a data value as a function of two or more input values. This is used in the context of determining which value to place in a gridded bin in a map projection. For example the result could be an average, maximum, minimum, or the result of a weighting function. A step-by-step problem solving procedure, especially an established, recursive computational procedure for solving a problem in a finite number of steps. Used in this document to represent a section of computer source code that performs this step. Used in this document to mean data that has been transformed into the MODIS scan cube data format. American National Standards Institute. The governing body for published standards, one of which is used to standardize programming language source code and specifications. Algorithm Transfer Team. A group within the MODIS SDST that is chartered with helping algorithm programmers with porting source code to the PGS environment. Used in this document to mean external data that is not in the MODIS scan cube data format. Advanced Very High Resolution Radiometer. An instrument on a series of NOAA satellites. See the "NOAA Polar Orbiter Data" by K. Kidwell, available from NOAA for details. A number used to represent a range of energy wavelengths from which a single detector measures a radiance energy. This is the same as a data channel for AVHRR and several other instruments but is not the same as a data channel for the MODIS instrument or other instruments with variable spatial IFOV coverages. A constant temperature emissive source within the MODIS instrument used to calibrate the thermal bands. The detector energy measurements when viewing the MODIS Black Body. A computer programming language, successor to the A and B languages, the name of which is consistent with the UNIX minimalist user command interface. Coastal Zone Color Scanner. An ocean discipline satellite remote sensing instrument. Distributed Active Archiving Center. The EOS component for formal Data Product storage, production, and access. 53 PAG_ _ NOT I_IUAED Data Product detector DCB Earth View ECS EOS Frames (MODIS instrumen0 FTN or FORTRAN geolocafion global header (2 defs) heritage hierarchical IFOV IMS A formal name of a science (or instrument) value that is registered with the SPSO database. A piece of hardware in the MODIS instrument that measures radiant energy at one IFOV. Data Control Block: a table that is used to maintain information about a dataset. This may contain file names, sizes, pointers, links to associated datasets, and dataset descriptors. MODIS instrument detector readouts during the portion of the scan in which the Earth is in view. EOS Core System. The ground processing and archiving, computer based segment of the EOS Earth Observing System. A suite of remote sensing satellites. A component of MTPE. All the MODIS instrument data corresponding to a nominal 1 kilometer section of an across track scan. (830 data channels) Short for Formula Translator. A computer programming language. The act of determining the location, on the Earth geoid, of MODIS instrument IFOVs. This is performed for each spatial element (1 kilometer nominal footprin0. Of, relating to, or involving the entire earth; worldwide. A section of the dataset containing information about the dataset (applied to both the dataset volume and each scan cube). Also, a section of source code that specifies data structures and descriptors that are common to more than one module. This library uses library include t'des, operating system include files, generic data structure include files, and Data Product specific header include files. Something that is passed down from preceding generations; a tradition. In the MODIS case, techniques and algorithms that were used in the past, before MODIS was invented. Of or relating to a hierarchy, a series in which each element is graded or ranked. In computer terms, elements of the hierarchy are wholly contained in (a subset of) higher elements. Instantaneous Field Of View. Used informally to mean the instrument field of view. This is the spatial area over which energy is measured by each instrument detector. Information Management System. A component of the ECS that performs data product selection based on metadata values. 54 Latitude Level-O Level-1A Level-lB Level-2 Level-3 Limb Longitude mapped MAS MCST Metadata MISR MODIS MTPE The angulardistancenorthorsouthof theearth'sequator,measuredin degreesalonga meridian,asonamaporglobe.Theoriginis attheEquator,increasingto 90degreesat theNorth pole. A formalDataProductdefinedto betheMODIS instrumentoriginaldatain packetform with CCSDS(ConsultativeCommitteeonSpaceDataSystems)headersprepended. A formalDataProductdefinedto betheMODIS instrumentdetectorvaluesasraw counts.This includestheEarthView, SolarDiffuser, SRCA,BlackBody,Spaceview,and engineering/ memory dump data. Level-1A is fully reversible to the Level-0 Data Product. A formal Data Product defined to be the MODIS instrument at-satellite detector values as radiance energy values received at the satellite in the instrument spatial geometry. This is not the at ground radiances and therefore has no atmospheric correction applied. Formal science Data Products in the instrument spatial geometry. Formal science Data Products in a globally mapped representation, corresponding to geographic shapes and dimensions. The circumferential edge of the apparent disk of a celestial body. Used in MODIS terminology as the outermost detector measurements at the edges of the scan (maximum scan angles). Angular distance on the earth's surface, measured in degrees east or west from the prime meridian at Greenwich, England, to the meridian passing through a position. Positive to the East, negative to the West and less than 180 degrees in magnitude. A representation, usually on a plane surface, of a region of the earth or heavens. The ECS will be using many types of maps produced via several gridding, binning, and aggregating schemes. MODIS Airborne Simulator. A 50 channel aircraft instrument with MODIS like bands. MODIS Characterization Support Team. The MODIS entity responsible for the calibration and characterization of the MODIS instrument. Data about other data. E.g., a synopsis of a Data Product used for selection criteria. An instrument on EOS with forward and backward looking scans used for stereo and bi-directional reflectance Data Products. Moderate Resolution Imaging Spectroradiometer. Mission To Planet Earth. NASA's project that focuses attention of the Earth as opposed to space or other planets. 55 multiplexed nadir nbands nBBframes nEMbytes nframes nlines npixels nl'es nSDframes nsize nSRCAframes Relating to, having, or consisting of multiple elements or parts. Relating to or being a system of simultaneous communication of two or more messages on the same wire or radio channel. A system in which data values are combined together, asynchronously. A point on the celestial sphere directly below the observer, diametrically opposite the zenith. The pierce point on the Earth of the MODIS instrument to center of Earth vector. The integer number of distinct wavelength bands for Level- 1A and Level- 1B products. Also used as the number of parameters in a Data Product. The number of instrument frames that contain detector views of the black body The number of equivalent instrument frames that contain memory dumps from the instrument and formatter onboard computer memories. All engineering data, intemal tables, and op codes are contained in these data dumps. The integer number of sampled MODIS instrument detector elements in the across track (along scan) direction for the finest resolution bands or parameter. This is unique to each Data Product, but would correspond to the number of pixels for the 250 meter bands for the MODIS Level- 1A and Level-lB Data Products and would have a nominal value of 5416. The integer number of lines of sampled MODIS instrument detector values in the along track (across scan) direction for the finest resolution bands or parameter. This is unique to each Data Product, but would correspond to the 40 pixels for the 250 meter bands for the MODIS Level- 1A and Level- 1B Data Products. The integer total number of sampled MODIS instrument detector values in both the along track (across scan) direction and the across track (along scan) direction for the finest resolution bands or parameter. This is unique to each Data Product, but would correspond to the 40 pixels times nframes for the 250 meter bands for the MODIS Level-lA and Level- 1B Data Products. The integer divisor applied to the nfames and nlines values for each band or parameter of the formal Data Product. This can be considered as an array of dimension [nbands]. For MODIS, this number = 1 for bands 1 and 2, - 2 for bands 3 to 7, and = 4 for the remaining bands. The number of instrument frames that contain detector views of the solar diffuser. The number of bytes that a data element consumes. This is the total for all components of the data element. For example, a Data Product consisting of a double, a long, and a byte value would consume 13 bytes per pixel. The number of instrument frames that contain detector views of the Spectral Radiometric Calibrator Assembly. 56 nSVframes OBC parameter PGS pixel prototype (ANSI) QA quantization radiance raw counts rectified rectilinear resampling scan SDPO SDST The number of instrument frames that contain detector views of deep space. OnBoard Calibrators. The set on four calibration sources contained within the MODIS instrument. See Solar Diffuser, SRCA, Black Body, and Space View. Used in the MODIS context to define a component of a Data Product. Several parameters constitute a formal Data Product. QA is included as a parameter to a Data Product. Product Generation System. A component of the ECS that produces Data Products. picture element. The smallest granule of a picture that can be represented by a unique numerical value, usually on a video display. A skeleton function call definition containing all the type declarations of its arguments, but without the actual variable names. Quality Assurance. The methods applied to MODIS Data Products for, and the resulting indicators of determining the quality and validity of the Data Product. To limit the possible values of [a magnitude or quantity] to a discrete set of values by quantum mechanical rules. The process in which a continuous physical value is divided into discrete values, thereby limiting the precision of the measurement. The radiant energy emitted per unit time in a specified direction by a unit area of an emitting surface. An integer number representing the amount of energy measured at a detector. This is sometimes called a digital number (DN) in other documentation. To set right; correct. To correct by calculation or adjustment. Used in the MODIS sense to mean a straightening of the scan geometry to orthogonal axes and uniform spatial sampling. Moving in, consisting of, bounded by, or characterized by a straight line or lines. The effort of resampling pixels in the MODIS instrument geometry with the 'bow tie' effect into a linearized and parallelized orbital coordinate system. A process in which a value at a prespecified location (spatial or temporal) is derived from the values of its surrounding values. The data acquired during one half of the scan mirror rotation, consisting of multiple instruments Frames, and assembled into a data structure called a scan cube. See the MODIS Data Structure, Volumes, and Rates document for details. Science Data Products Office. MODIS Science Data Support Team. The GSFC based group responsible for the production component of the MODIS ground processing system. 57 SolarDiffuser SpaceView spatial Spatial Element SRCA swath TBD temporal TM TM token typedef (C) UNIX An onboard calibration white target, illuminated by the Sun. The MODIS detectors view this target during a portion of the mirror scan rotation. MODIS instrument detector readouts during the portion of the scan in which the deep space is in view. Of, relating to, involving, or having the nature of space, reladng to geometry as opposed to time (temporal). The nominal (at nadir) 1 kilometer area covered by an ideal MODIS IFOV. Each element contains all bands of data, irrespective of spatial resolution, that occur at each 1 kilometer footprint. See the MODIS SRR, PDR, and CDR documents for exacting details. Spectral Radiometric Calibrator Assembly. The contiguous area viewed by a MODIS instrument scan segment. For example, the portion of the MODIS mirror scan that views the Earth. To Be Determined Of, relating to, or limited by time: a temporal dimension; temporal and spatial boundaries. Thematic Mapper. A satellite remote sensing instrument. Team Members. Members of the MODIS Science Team. A string of characters separated by a delimiter, usually a blank character (or whitespace in UNIX terms). A "C" language facility that allows the programmer to define a new, possibly compound data type. Examples of predefined data types are int, long, short, float, and double. The computer operating system selected by EOS as the standard platform. 58 REFERENCES Unless otherwise noted, these documents are available from the MODARCH data archiving facility, code 920, Goddard Space Flight Center, Greenbelt, Maryland. This facility is available via the Intemet. 1 - "MODIS Level 1A Software Baseline Requirements", SDST, Sept. 1, 1993, NASA TM 104594, Vol 1. 2 - "MODIS Data Rates, Volumes, & Processing Performance w/Data Structures", Thomas Goff, May 13, 1994. 3 - "MODIS Processing Spatial Domains", Virginia Kalb and Thomas Goff. 4 - "MODIS PGS Data Processing Operations Concepts,", SDST, Sept. 13, 1993. 5 - "Science Computing Facilities Plan", Edward Masuoka, Sept. 7, 1993. 6 - "Level 1A System Requirements Review (SRR)", SDST, May 11, 1993. 7 - "PGS Toolkit Users Guide for the ECS Project", EOSDIS Core System Project, Feb. 4, 1994. 8 - "MODIS Software Development Standards and Guidelines", SDST, May 3, 1994 (draft). 9 - "Geometric Correction of MODIS Data", Virginia Kalb and Thomas Goff, Sept. 23, 1993. 10 - "MODIS Level 1A Earth Location ATBD", Robert Wolfe, July 25, 1994. 11 - "MODIS Sensor Patterns and Multiresolution Pixel Registration", A1 Fleig, luly 27, 1994. 12 - "EOS Reference Handbook", Earth Science Support Office, Document Resource Facility, 300 D Street NW, Suite 860, Washington D.C. 20024. 13 - "The Moderate Resolution Imaging Spectrometer-Nadir (MODIS-N) Facility Instrument", Advances in Space Research vol 11(3), 231-236, 1991, V.V. Salomonson and D.L.ToI1. 59 (Thispageintentionallyleft blank) 60 INDEX A ancillary data, 1 ANSI, 6 All', 10 AVHRR, 2 B bow tie, 2 C C7__S, 3 D data product header include file, 4 F FORTRAN, 8 G Geolocation, 1 L Landsat, 2 Level-lA, 10 Level-lB, 7 Level-2, 10 M make f'de, 8 mapped domain, 1, 2 MAS, 3 MISR, 2 MOD_IO_allocateOutputB uffer, 13 MOD_IO_ closeDatasets, 13 MOD_IO_closelnputDataset, 13 MOD_IO_closeOu tputDataset, 13 MOD_IO openMasterlnpu tDataset, 12 MOD_IO_openOutputDataset, 12 MOD IO...printDataDescriptor, 26 MOD_IO_prototypes.h, 9 MOD IO readSwath, 12 MOD_IO_writeSwath, 13 MSS, 3 61 PiaWall_m4 P4_d_ _ PlOT _, _-G-+ +; i " ' ' " FIL.M -EJ_ Nnadir, 32 NOAA, 4 p i PGS, 9 POSIX, 8 pseudo code, 11 R ragged array, 4 rectilinear domain, 1 resampling, 2 S SBRC, 10 scan cube domain, 2 SDST, 8 spatial registration, 9 T TLCF, 17 TM, 3 typeaef, 9 7 U UNIX, 17 i V vegetation index, 7 VI, 7 62 Form Approved REPORT DOCUMENTATION PAGE OMBNo.0704-0188 Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorafe for Information Opera(ions and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188), Washington, DC 20503. 1. AGENCY USE ONLY (Leave b/ank) I 2. REPORT DATE 3. REPORTTYPE AND DATES COVERED I November 1994 Technical Memorandum 4. TITLE AND SUBTITLE 5. FUNDING NUMBERS MODIS Technical Report Series Volume 4, MODIS Data Access User's Guide - Scan Cube Format 6. AUTHOR(S) Virginia L. Kalb and Thomas E. Goff 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS (ES) Goddard Space Flight Center Greenbelt, Maryland 20771 9. SPONSORING ! MONITORING ADGENCY NAME(S) AND ADDRESS (ES) National Aeronautics and Space Administration Washington, DC 20546-0001 Code 920 C-NAS5-31331 8. PEFORMING ORGANIZATION REPORT NUMBER 95B00013 10. SPONSORING / MONITORING ADGENCY REPORT NUMBER NASA TM-104594, Vol. 4 11. SUPPLEMENTARY NOTES Kalb: Goddard Space Flight Center, Greenbelt, Maryland; Goff: Research and Data Systems Corporation, Greenbelt, Maryland 12a. DISTRIBUTION /AVAILABILITY STATMENT 12b. DISTRIBUTION CODE Unclassified - Unlimited Subject Category 61 This publication is available from theNASA Center for AeroSpace Information, 800 Elkridge Landing Road, Linthicum Heights, MD 21090-2934, (301) 621-0_90. 13. ABSTRACT (Maximum 200 words) The software described in this document provides I/O functions to be used with Moderate Resolution Spectroradiometer (MODIS) level 1 and 2 data, and could be easily extended to other data sources. This data is in a scan cube data format: a 3-dimensional ragged array containing multiple bands which have resolutions ranging from 250 to 1000 meters. The complexity of the data structure is handled internally by the library. The I/O calls allow the user to access any pixel in any band through "C" structure syntax. The high MODIS data volume (approaching halfa terabyte per day) has been a driving factor in the library design. To avoid recopying data for user access, all I/O is performed through dynamic "C" pointer manipulation. This manual contains background material on MODIS, several coding examples of library usage, in-depth discussions of each function, reference "man" type pages, and several appendices with details of the include files used to customize a user's data product for use with the library. 14. SUBJECTTERMS MODIS; Scan Tube 17. SECURITY CLASSIRCATION OF REPORT Unclassified NSN 7540-01-280-5500 18. SECURITY CLASSIRCATION OF THIS PAGE Unclassified 19. SECURITY CLASSIRCATION OF ABSTRACT Unclassified 15. NUMBER OF PAGES 72 16. PRICE CODE 20. LIMITATION OF ABSTRACT UL Standard Form 298 (Rev. 2-89)

MODIS AOD mapping Matlab file

Abstract

Extracted data

MODIS AOD mapping Matlab file

Abstract

Extracted data

Related items

Related items