A basic knowledge on computational programming is a very important resource for the modelling of natural systems. Following the entry on the "Water Tohoku: Tutorials" in reference to C programming and GCM data retrieving, in the current entry I cite and shortly review some tools useful for viewing and editing of large multidimensional matrices.
Why do we need to deal with large matrices?
The growth in the power of computational processors (supercomputers) currently permits the investigation of complex natural systems. The research in the fields of climatology and meteorology (in reference to the natural sciences) has undoubtedly given a good use on such technological advances; consequently, the growth in the number of research centers throughout the world (e.g., the Center for Climate System Research at the University of Tokyo) has grown as well.
The extensive amount of data generated from the investigation of such complex natural processes has high practical relevancy, because of which the necessity to deal with large files is becoming an issue to deal with every day.
Climate change is perhaps the most relevant field where climate research outcomes are to be used. The investigation carried by the Intergovernmental Panel on Climate Change IPCC provides tangible examples on such relevancy. The IPCC provides several resources to climate change researchers. Particularly, water resources scientists may be especially interested in the analysis of socioeconomic and environmental data produced by climate models, for research and assessment. That data has been gathered by the IPCC Data Distribution Centre DDC, and is provided at different resolutions. The issue is that such data is quite large, it is provided in a format commonly managed by meteorologists and climatologists, for instance it is almost impossible to open and view using common utilitarian software (e.g., Microsoft excel, notepad, etc). Similar issues may be found when dealing with distributed hydrological models (whose complexity is minimum compared to climate models). For the cases described above, the following resources will be very useful.
Viewing and processing large matrices.
1. C and Fortran compilers:Very useful and well known commercial software (e.g., Matlab, Mathematica) is currently available for viewing large matrices. However, commercial software packages are not a valid option in several educational institutions, principally because the acquisition cost is not low compared to the cost of C and Fortran compilers.
- Requirements: Basic knowledge on C, Fortran. The format of the input must be well understood beforehand.
- Advantages: Accessible cost, processing and calculation speed is very high compared to Matlab and Mathematica.
- Disadvantages: Sometimes the format of the climate model outputs is not fully detailed in their correspondent "readme" file (e.g., data in int.?, float?). Other times, the data to be read may have a complex format. As result, the data read through the C and Fortran compilers may not be correct.
2. Transform: This software was developed by the USACE, and commercialized by Fortner. I prefer using this to read ASCII and binary matrices, which later I save in binary format to produce the inputs for my C program.
- Requirements: None (easy to use).
- Advantages: It will automatically detect the number of rows, columns, and data type of the file.
- Disadvantages:Processing is very limited (it is not a compiler). I could not get a link to the developer.
- Requirements: None.
- Advantages: Free. It can recognize several formats (including HDF).
- Disadvantages: The processing of the data may not be easy to understand.
4. UltraEdit Text Editor: It is a powerful text editor. I have used it to open very large text files, and then to extract a portion of the data for further processing.
- Requirements: None.
- Advantages: Very easy to use. It can manage very large files.
- Disadvantages: It is not free. It may not open certain data formats (e.g., HDF).
No comments:
Post a Comment