Exporting, importing and projecting data layers with DIVA-GIS

A United Nations Development Programme (UNDP) Global Environmental Facility (GEF) project covering the Lake Tanganyika riparian countries, Burundi, the Democratic Republic of Congo, Tanzania and Zambia.

In this lesson you will learn how to import and export data from DIVA-GIS to other GIS software and applications; including a short introduction to ArcView and using the dataset in ArcView; you will also learn how DIVA-GIS can project data from Geographic coordinates to other projections.

DIVA-GIS can be used for managing data sources, and for exporting and importing data from and to DIVA-GIS. Start DIVA-GIS and create a new empty project (e.g. lesson7 in your my diva folder). Add the hotspot grids you created in lesson 6 to the project.

To rename or permanently remove files from your computer use the File Manager, available under the menu: Data – File Manager,

. The file manager can handle grid files, shape files, and Diva project and export files.

Note that if you delete files using the File Manger, they will not be moved to the Recycle Bin in windows, but will be permanently deleted. If you rename or remove a file that was included in a project, the project will no longer have access to the file, and it will be omitted when you open the project again.

For grid files there is a special tool for retrieving statistical data about grid layers and for managing grid data headers, under the menu: Grid – Describe,

. Apart from getting statistical information, this tool can be used to update erroneous grid file header (*.grd – see lesson 5), but should only be used for such purposes if you are familiar with data types and there is an error in your grid file. The Grid describes is further very slow, and if you have a large grid it can take more than 10 minutes to load the Grid describer.

DIVA-GIS can import both vector and grid data, reached under the menu: Data – Import Points to Shapefle / Import Text to Line/Polygon. For vector data it can be a useful tool for importing point data with X and Y coordinates, including data obtained with a GPS receiver. DIVA-GIS can import data with coordinates from either text files (*.txt), dBase files (*.dbf) or from Microsoft Access (*.mdb). Text files (*.txt) can be imported either as points or as lines/polygons.

Data captured with a GPS device can be added to DIVA-GIS in several ways. Many GIS devices come with software that can transform the captured Waypoints (WP) and tracks to shape files, that can then be used directly in DIVA-GIS. If this is not possible you can always save WP and tracks as simple text files, with the data delimited by tab, semicolon, comma or space using the software that comes with any GPS device. If you save you GPS data as a text file DIVA-GIS can read it and convert it to a shape file for you. Go via the menu: Data – Import Points to Shapefile, and choose the text file with GPS data as Input File and give an Output File name. In the Create Shapefile from Text File window you must click the radio button for the correct Field delimiter (Comma in the example below), and DIVA-GIS will get the data organized for you. A nice option in DIVA-GIS is that when importing a points from a text file you can specify the database to build from the text file. Click in one of the cells, as shown below, and you can change the default format for each field. In most cases you will not want to change the format, but it can be useful to change the size of a text field or a misinterpreted field (e.g. if there are text qualifiers around numbers)..

When you have defined the entries to the text file to import, click Apply, and DIVA-GIS will generate a shape file from the waypoints for you.

If you want to import a track from a GPS, you can also do that. The text file must then have a more specified format, as shown below. Each row in the text file should have a coordinate pair (X and Y or longitude and latitude, as in the example). The first row should also start with an identifier before the coordinate pair. There can be more than one line or polygon in the text file, with a single END on the row finishing one line or polygon, and the next row starting a new polygon. At the very end of the text file there must be two rows with END following each other.

With a text file formatted as in the example above you can use DIVA-GIS to create either lines or polygons in shape file format. Go via the menu: Data – Import Text to Line/Polygon, use the drop down menu to select if you want to creates lines or polygons, then get the Input File and give a name for the Output File.

Click Apply to create the vector shape from the formatted text file. If the checkbox for Add to map was checked the new line (polygon) will be automatically added to the DIVA-GIS project.

You can also digitize your own vector files directly on the screen using DIVA-GIS, from the menu: Data – Draw Shape. You can digitize points, lines or polygons, and save the data to a shape file (*.shp). In this way you can create a vector layer identifying e.g. land degradation hotspots (the example used here).

You can digitize several points, lines or polygons in one digitizing session, closing a line of polygon by clicking the Stop Drawing button,

, or double clicking when you digitize the last point of a line or polygon feature. You can also add values and labels in the Draw Shape window, as shown below.

Save the digitized features by clicking the Save button,

, in the Draw Shape window, and the new vector will be added to the DIVA-GIS project. You can then symbolize the new shape, as illustrated in the example below.

DIVA-GIS can convert data from grid to vector, and vice versa. The polygon to grid conversion was introduced in lesson 5, and is a good tool. The vector to grid conversion is a bit cruder, but is useful when e.g. projecting a grid (see below). The exported shape file will consist of as many features as there are cells in the grid file. To export a grid file to a vector file use the menu entry: Data – Export Gridfile, then choose either to generate Shapefile (polygons) or Shapefile (points), as shown below.

From the illustration above you can see that each cell from the grid is exported as a feature in the new shapefile (points in the example). The generated shapefile will hence have as many features as there are cells in the grid file, and will be very slow to work with (if you try to symbolize this vector file it will take a very long time for DIVA-GIS to process it).

DIVA-GIS can import and export grid files in several formats, including files used by Idrisi (Kilimanjaro), ArcView/ArcGIS, and general grid files (BIP, BIL and BSQ – explained below). The grid file import/export functions are under the menu: Data – Export Gridfile / Import Gridfile, and allows export/import of multiple files in one go.

There are more options for exporting compared to importing, but the conversions that DIVA-GIS can do are sufficient for sharing data with almost any other GIS software. If you want to use the data in the dataset with e.g. Idrisi or ArcView, you can simply choose Data – Export to GridFile – Multiple Files, select all files in a folder and export them (to the same folder or another folder). First the interface for exporting a single grid file for use in Idrisi.

The data exported to Idrisi (or GRASS, which is a very powerful GIS software available for free, but more difficult to use) can be directly used in Idrisi (GRASS) once exported.

You can also export grid files for use in ArcView/ArcGIS (produced by ESRI – Earth System Resources Incorporated). There are two options for exporting data to ESRI (ArcView/ArcGIS), as ESRI binary (FLT) or as ESRI ASCII. The former is better for real data (see lesson 5 to learn about data formats in grid files), and the latter for byte or integer files. In the example below the two hotspot maps that you created in lesson 6 are exported to ESRI ASCII format. ArcView grid names can only be ten characters, and you can click the checkbox Make valid EDRI grid names to truncate the names of the exported gridfiles. In the tutorial the names will be changed when imported to ArcView instead. Click Apply to export the listed files.

Data exported to ESRI formats are generated in a special exchange formats used by ArcView/ArcGIS, and must hence be imported to these programs before they can be used.

This section will deal with importing grid (e.g. exported from DIVA-GIS) to ArcView. To follow this section you must have an ArcView license, including the extension Spatial Analyst.

Start ArcView, and open an empty View. First you must install the extension for Spatial Analyst (the module that handles grid data in ArcView) from the ArcView menu: File – Extensions.

In the Extensions window, scroll down and click the checkbox for Spatial Analyst (if you do not have the extension Spatial Analyst you will not be able to use grid data with ArcView, but you can still all the datasets in the dataset by using the jpg images instead). It is not enough to click the text, you must click in the checkbox.

Click OK for ArcView to load the extension. Make sure the View is the active document in ArcView (the frame of the active document - View in this case, appears dark blue by standard). Then go to the import function via the menu: File – Import Data Source, as shown below.

Dependent on whether you exported an ASCII file (*.asc) or a binary file (*.flt) from DIVA-GIS you have to choose the corresponding file type in the Import Data Source window of ArcView, as shown below.

In the Import window that opens navigate to the folder where you put the exported grids from DIVA-GIS, and select the files you want to import, you can import multiple files in one go (hold down ctrl+alt while selecting), as shown below.

Click OK to start the import to ArcView. In the Output Grid window that opens you can navigate to the folder where you want to put the data, and give a name you want the ArcView grid to have.

If you Imported ASCII files to ArcView you will be prompted if you want Cell values as integers, which is preferred if the exported data was either byte or integer (see lesson 4). If the data type was real (called FLT4BYTES in DIVA-GIS) you should not save the Cell values as integers.

You can then choose to add your data to the View that was the active document when you started the import.

To add other data to ArcView, including the shape files (which is a format invented by ESRI and the default format for ArcView) and jpg image files, use the Add Theme tool,

, in ArcView. To add vectors you must set the Data Source Types: Feature Data Source in the Add Theme window, as shown below.

To add the jpg files you must change the Data Source Types: to Image Data Source (and to add grid files it must be set to Grid Data Source).

In the example below the image data source (*.jpg) for the normalized trend in increment RNNDVI, and the Feature data source (*.shp) for country borders have been added to ArcView.

Both DIVA-GIS and ArcView can handle the general BIL (Byte Interleaved by Line) data format. This is a file format where the data is stored row by row (row = line) for images with multiple bands (more than one data layer in one file) as one long string of binary values. In comparison BIP (Byte Interleaved by Pixel) files store data values pixel by pixel. And BSQ stores that data Band SeQuential (first band one, then band two etc). If there is only one band in the file (as is the case for all data in the dataset of this project) BIP, BIL and BSQ files are exactly the same, it is only if more than one data layer is stored in the same file that the format becomes different. In DIVA-GIS you can convert any grid file to BIL format for use in other image processing software (idrisi, ArcView, ArcGIS, ENVI, Ermapper etc). All rainfall, vegetation and RNNDVI grids available in the project dataset can be easily converted to BIL or Idrisi format by using windows batch files provided in the dataset – under the folder \sahel11\sahel\log. This is explained in detail in the next section of this lesson.

For other data (e.g. the grid files you created on your own, and grid files over land cover, vegetation fields and population that you used in other lessons) you must use DIVA-GIS to convert the DIVA grid file to other formats manually. Return to DIVA-GIS and export the landcover map (\data_spatial\sahel\grid\landcover\sahel_igbp_lc) to a BIL file, as shown below (you can put the exported file in the same directory as the original file, not in the my diva folder).

Go back to ArcView and open the exported BIL file via the Add Theme tool,

, navigate to the folder with the exported BIL files, and add them as images. As there are both jpg files (included by default in the dataset – you looked at them in several of the lessons before) and BIL files in the folder after the export from DIVA-GIS you must be careful in selecting the correct files. ArcView will accept both the jpg image and the BIL file, but will only understand the information content in the BIL file, the jpg file will only have color values (no values related to e.g. landcover, vegetation or rainfall). Both files can be added by setting the Data Source Types: Image Data Source as shown below.

Below is the landcover BIL map that you created added to ArcView, but because the BIL file is in integer format where values can be in the range –32768 ..32767, and the values in the landcover grid is only between 0 and 16, the symbolization does not work. ArcView automatically assigns a stretched black and white symbolization for all BIL files, and if data is just occupying a small range in a BIL file with integer data, the content is not seen.

To work with the added grid in ArcView (and in our case to see the data at all), the BIL file must be saved as grid file in the ArcView default format. To save the BIL file as grid go via the ArcView menu: Theme – Convert to Grid.

To make things even more complicated ArcView can not import real data in BIL format. The trend grid files as well as all the files associated with the trend (mean, standard, deviation, correlation and significance grid files) that we used (and generated) from DIVA-GIS are all in real data format and must hence be exported and imported as binary (*.flt) files when converted from DIVA-GIS to ArcView (see above).

ArcView and DIVA-GIS also share the concept of a project. Both programs store map compositions, legends and design (called layout in ArcView) in project files (*.div for DIVA-GIS, *.apr for ArcView). The datasets for this project was generated using ArcView. When the dataset was created, ArcView was programmed to store all the information in a project (.apr) file. If you have an ArcView licence you can hence open the complete dataset by opening the project ready made ArcView project file. The ArcView project file comes in two versions, one containing all framework datasets, and one containing only vector framework datasets. The former project file demands that the extension Spatial Analyst is installed; otherwise the grid files can not be loaded. The latter version (with the suffix _nogrid) is for those who have a basic ArcView license, and does not demand Spatial Analyst.

Use Windows explorer (or start ArcView and at startup choose Open an existing project) and navigate to the locations indicated in the table above, and double click one of the .apr files. If you have the extension Spatial Analyst installed open the project without suffix, if you lack the extension Spatial Analyst instead open the project file with the extensions _nogrid. There are some free extensions added on the Project CD, under the folder \software\arcview. These extensions have different functions that can be added to ArcView by putting them in the default extension folder of ArcView GIS (with a normal ArcView installation the folder to copy these files to is C:\ESRI\AV_GIS30\ARCVIEW\EXT32). With the extension JFIF.avx added you can view all the data layers (including all the satellite derived vegetation and rainfall datasets) in ArcVew by adding them as images (see above). You will however not be able to use these images for any map calculations.

When the Sahel dataset was compiled, several windows script (or batch) files (with the extension .bat) were generated that can be used for converting all the rainfall and vegetation grids to Idrisi or BIL format. Above you used the BIL format to import files to ArcView, but most other image processing software can also import BIL files. If you look in the folder \data_spatial\sahel\log, there are many .bat files for automatically converting the DIVA grid files to other formats. Just double click on the bat file to execute it. There are bat files for both copying and renaming files. If you choose to copy files you must make sure that you have enough disk space for copying. If you rename the files you do not need any additional disk space, but the files must be renamed back to the original names before you can use them in DIVA-GIS again (bat files for getting back the the DIVA-GIS name are also available). The naming of the bat file reveal which data layers the bat file operate on. The bat files will keep the files in the original source folders, as specified in the web-catalog and as used in this training module. The table below summarizes the bat files available for converting the rainfall and vegetation data for use in other software than DIVA-GIS

Projecting layers

All the data on the project CD are in geographic (longitude – latitude) coordinates. The geographic system is not a true projection, but with DIVA-GIS you can project your vector layers to almost any other projection. Unfortunately DIVA-GIS can not project grid files, but if you need to project a grid file you can first export it to a Shapefile (polygon), then project, and then convert it back to a grid. In DIVA-GIS you can not project an already projected layer.

Use the Add Theme tool to add the vector line that shows the country borders (\data_spatial\sahel\mapdata\politic\sahel_generalized_borders.shp). With the layer added, from the DIVA-GIS menu select: Tools – Projection.

In the Projection window select sahel_generalized_borders as Input (default if it is the active theme in the Legend). Select an Output file name and at the same time create a new folder to put your projected data in (e.g. sahel_utm under \data_spatial as in this tutorial). Give the Output File the same name as the Input File but add the ending utm30. Make sure the tab for Standard is selected in the Projection window (see below), click the drop down menu for Category and select UTM – 1983. Then click the drop down menu for Type and set it to Zone 30. The standard projection parameters for UTM 30 will be shown in the Projection window as illustrated below.

You only need to click the Apply button,

, and DIVA-GIS will project the layer for you. The new layer will be automatically added to the Data View, but if you check the checkbox for the projected layer, it will not be seen. To see it make it the active theme and click the Zoom to Theme tool,

. The Data View zooms to the projected layer, and you will see how the map of West Africa is distorted, epically towards East. In the UTM projection a true scale is only retained along the central longitude (or meridian) in this case along the meridian –3 (3 degrees West). The UTM projection is commonly used by smaller countries, or countries with a North-South orientation. Larger countries usually have other standard projections. UTM zone 30 is not useful in for instance Niger (too far East for UTM 30) or Senegal (too far West for UTM 30).

If you look at the lower left of the Project window you can see the coordinate information for the map in the Data view, and that it has changed compared to the original data in geographic coordinates.

You can check the coordinates and scale out by toggling between the projected and the geographic layer, most easily done by using the Zoom to Previous Extent,

, and Zoom to Next Extent,

, buttons. Note how the Scale changes despite the Data View looking similar. The projected data is better at depicting true distances, areas and angles (depend a bit on what projection you are using).

The standard UTM zones use the spheroid GRS80 to describe the roundness of the Earth (the Earth is a bit flattened due to its rotation, and can not be described as a perfectly round sphere, but one that has a larger horizontal radius compared to the vertical radius). More modern data, however, use the WGS84 spheroid (the difference is small as you will see, but for other spheroids that might not be so). To project the Sahel country borders to UTM 30, but using the WGS84 spheroid, make sure the original layer (in geographic coordinates) is the active theme, and again open the Projection window (form the menu: Tools – Projections). This time click the Custom tab, set projection to Transverse Mercator in the Projection drop down menu, and the spheroid to WGS84 in the Spheroid drop down menu, as shown below.

You then have to fill in the projection parameters yourself shown below (as you want to use the same parameters as for the Standard UTM30 zone you can toggle to the Standard settings to find the parameters by using the Standard and Custom tabs).

Click the Output button and give a name, but this time add the suffix utm30-wgs84, as shown above. Click Apply and DIVA-GIS projects the Sahel country borders again.

You can now compare the standard and the custom projections you have done. If you turn the layers you projected on and off, you will see that they fit perfectly. Even if you zoom in you will not see any difference between the layers. This is because West Africa is close to the equator and the description of the roundness of the earth has a very small influence when projecting maps, further away from the equator the different spheroids will generate slightly different results.

You can use the projection tool in DIVA-GIS to project data from geographic coordinates to almost any projection, but you can not project data from one projection to another, or back to geographic coordinates. As all the datasets that comes with the project are in geographic coordinates you can however change them into your preferred projection, and then combine the supplied datasets with other data that you have available, even if they are projected.

DIVA-GIS does not contain any function for directly projecting a grid layer from geographic coordinates to other projections. It is a rather complicated process to project a grid as you must first convert it to a shape polygon, then project the polygon, then create a template to set the new grid parameters manually, and finally convert the projected polygon back to a grid. But it is possible to do. In general it would be better to do any grid calculation using the original (geographic) data, and then just extract the key information and convert the grid to a shape file. You can then apply the Select Feature tool to the exported shape file to extract only the features you are interested in and Save the selection as a shape file (see lesson 3). Then project the shape file with your selected feature as shown above. You can also use the digitizing options, and just project the digitized features to your preferred projection.

To project a grid file you must first Export it to a polygon shape file, as done above in this lesson. In the example below, the gridded land degradation hotspot layer derived from the Multi Criteria Evaluation (MCE) was exported to a polygon.

Exporting grids can be slow, but all the information in the grid will be converted to a shape file containing as many features as there are cells in the grid. To symbolize this vector file takes a long time as it consist of as many records as there are cells in the grid, as you can see in the zoomed Data View showed below.

To project a file is a heavy calculation, hence you should first select the records (rows) in the attribute table that you want to export. In this case you only want to export the hotspots, and other polygons can be omitted. To select the polygons identified as hotspots, you should use the Select (Query) tool,

. In the Select window, click the Select by Query tab. Under the Select by Query tab click the drop down menu for Field and select the VALUE field, then enter the criteria, less than “<” in this case, and enter the Value to be –0.1 (querying for the more degraded areas). Then click the Add button,

, and the query will be shown in the main area. You can add more conditions (using other fields) and build very advanced queries in DIVA-GIS, but here this simple query is enough. Click Apply to perform the query.

You should now have polygon files with as many polygons as the grid file it was derived from has cells, and with only those polygons identified as potentially degraded selected.

To project the land degradation hotspot, make the shape file with valid records for hotspots the active layer (mce_hotspots_only.shp), then open the Projection window (via the menu: Tools – Projections). Set the projection parameters in the Projection window (as done above) and click the Apply button to project the polygons.

If your shape file contains many records the projection will take a (very) long time (depends on the power of your computer). The projected shape will be added to the Data View, and if you look carefully (zoom in) you will see that the polygon boundaries are no longer exactly horizontal and vertical (as they are in the geographic coordinates), but they are a bit skewed.

Before you can convert the projected polygon to a grid, you must first create a grid template for DIVA-GIS to use for the conversion. The template defines the projection, and number of columns and rows. First you have to make a copy of the original grid file you wanted to project. It is better to do it using windows explorer. Open Windows explorer, navigate to the folder with your original grid, and copy both the file with the extension .grd and .gri to the folder with your projected data (sahel_utm in the tutorial). Then rename both files to e.g. grid_template_utm30. (You can copy the files using DIVA-GIS by first exporting the grid to e.g. Idrisi (see above) and then import the Idrisi grid again to DIVA-GIS and then put in the folder where you want it).

You now have a copy of the original grid file, and you must turn it into a template for your new projection. You have to do a bit of editing using any text editor. The best option is to use the Notepad text editor that comes with Microsoft windows. To open it go via the Windows desktop: Start – Programs – Accessories – Notepad. In Notepad go via the menu: File – Open, and navigate to the folder were you saved your template for UTM30 (i.e. the copy of the original grid file that you want to project). In the bottom of the Open window from Notepad change File of Type: All Files, as shown below. Then open the *.grd (the GRD file contains the header for the grid data file, which has the same name, but with the extension GRI, you can see both files in the Open window below).

Open the *.grd file for the template that you created (grid_template_utm30.grd). You will then see the content of the standard DIVA-GIS header file for grid data., also shown below.

The content of the header file is described in detail in lesson 5. You now have to edit the header manually in order to turn it into a template for the new grid you want to make. You must change the projection, extent and resolution of the header to reflect the projected data.

With Notepad still open go back to DIVA-GIS and open the Properties window for the projected country shape file, e.g. by double clicking it in the legend. You have to use a projected shape file that has the same extent as the grid template file you want to create (in this case we use the whole Sahel map area).

The third row of the properties window contains the information you want: the extent of the projected grid,

. The four numbers refers to MinX, MaxY, MaxX and MaxY in the projected file. Copy those numbers to the grid header (in notepad) you want to use as a template. You can use the windows copy – paste functions to do it (e.g. Ctrl-c to copy and Ctrl-v to paste). Then also make the following changes in Notepad to reflect the new projection.:

Finally you have to calculate the resolution of the template grid in the projected units (meters in this case), as

Below is an example of how to do it using Microsoft Excel, also showing the formula you have to enter. The Rows and Columns you get from the old grid (notepad), the MinX, MaxX, MinY and MaxY you get from the DIVA properties window, and Xresol and Yresol must be calculated in excel.

Enter the calculated X-resolution and Y-resolution into the GRD file using Notepad. The Georeference section of the GRD file should now look like the example below.

Your template is now finished, hence save the edits you did in Notepad (press ctr-s or go via menu: File - Save).

When you have created a template you can convert the projected polygon vector back to a grid file. Make the projected hotspot file (mce_hotspots_utm30.shp) the active layer, and go via the menu: Data – Polygon to Grid. In the Polygon to Grid window, Input file is defaulted to the active shape file (otherwise click the Input File button and select the correct file – the projected hotspot polygon layer). The field to use for grid values is the VALUE field. Then set the Existing Grid for Parameters to the template that you just created, and give a logical Output Grid file name (mce_hotspots_utm36.grd in the example below). Click apply to convert the hotspots back to a (projected) grid.

Projecting a shape file can be done very exact, but projecting a grid always leads to a loss in geometric accuracy (true for all GIS software). In the example below you can see the difference between the projected shape file and the projected grid file. There is a small mismatch between the shape and grid layers that was created first by our definition of template, and secondly by the fact that the polygons are not perfectly straight (horizontal and vertical) when projected. Nevertheless you now also know how to project grid data to almost any projection using DIVA-GIS.

Extension	Function
Afrcver.avx	customized functions for the Africover land cover layer data
JFIF.avx	for handling jpg images in ArcView
mrsid.avx	for handling mrsid images in ArcView
tiff.avx	for handling tiff images in ArcView

Bat process	Filename	Example
Copy grid from DIVA format to BIL format	copy_gri2bil_sahel_”dataset”	copy_gri2bil_sahel_precip_1930-1995-mm.bat
Copy grid from DIVA format to Idrisi format	copy_gri2rst_sahel_”dataset”	copy_gri2rst_sahel_rnndvi-max_1982-2006.bat
Rename grid from DIVA format to BIL format	rename_gri2bil_sahel_”dataset”	rename_gri2bil_sahel_precip-sum_1982-2006.bat
Rename grid from DIVA format to Idrisi format	rename_gri2rst_sahel_”dataset”	rename_gri2rst_sahel_ndvig-average_1982-2006.bat
Rename grid from Idrisi format to DIVA format	rename_rst2gri_sahel_”dataset”	rename_rst2gri_sahel_precip_1995-2006-MM-DD.bat

Original entry	New entry
Projection=	Projection=UTM30
Datum=	Datum=WGS84
Mapunits=	Mapunits=METERS