Protocols for Data Sharing within FLOODsite

(1)

FLOODsite is co-funded by the European Community

Sixth Framework Programme for European Research and Technological Development (2002-2006) FLOODsite is an Integrated Project in the Global Change and Eco-systems Sub-Priority

Start date March 2004, duration 5 Years Document Dissemination Level

PU Public PU

PP Restricted to other programme participants (including the Commission Services) RE Restricted to a group specified by the consortium (including the Commission Services)

CO Confidential, only for members of the consortium (including the Commission Services) CO

Protocols for Data Sharing within FLOODsite

Report Number

T32-05-02

Revision Number 1_2_P01

Integrated Flood Risk Analysis

and Management Methodologies

Date

November

2005

(2)

FLOODsite Project Report Contract No:GOCE-CT-2004-505420

T32_05_02_DataProtocols_D32_3_v1_2_p01.doc 7 June 2006

ii

D

OCUMENT

I

NFORMATION

Title Protocols for Data Sharing within FLOODsite

Lead Author Mark Morris / Andreas Kortenhaus

Contributors Keiran Millard (HRW); Nokis project team.

Distribution Public

Document Reference T32-05-02

D

OCUMENT

H

ISTORY

Date Revision Prepared by Organisation Approved by Notes

03/08/2005 (2_0_P01) Mark Morris HRW Data protocols summary v2_0_p01 03/08/2005 (1_3_p03) Andreas

Kortenhaus LWI NOKIS manual version 1_3_p03 1/11/2005 1_0_p01 Mark Morris HRW Integration of data protocols summary

and Nokis manual within overview report on data protocols.

5/11/2005 1_1_p03 Andreas

Kortenhaus LWI Comments / edits

7/06/2006 1_2_p01 J Bushell HRW Formatting; addition of deliverable details on cover; change of distribution to PU

A

CKNOWLEDGEMENT

The work described in this publication was supported by the European Community’s Sixth Framework Programme through the grant to the budget of the Integrated Project FLOODsite, Contract GOCE-CT-2004-505420.

D

ISCLAIMER

This document reflects only the authors’ views and not those of the European Community. This work may rely on data from sources external to the FLOODsite project Consortium. Members of the Consortium do not accept liability for loss or damage suffered by any third party as a result of errors or inaccuracies in such data. The information in this document is provided “as is” and no guarantee or warranty is given that the information is fit for any particular purpose. The user thereof uses the information at its sole risk and neither the European Community nor any member of the FLOODsite Consortium is liable for any use that may be made of the information.

(3)

S

UMMARY

With 36 partners undertaking 35 different tasks across the five year duration of FLOODsite, a considerable amount of data will be required and generated. The nature of this data will vary from task to task and may include survey data, images, video, drawings, reports, papers, modelling data etc. As an Integrated Project, many of the tasks rely upon other tasks and feed results through to later research. It will be important that team members are easily able to identify, access and use the various forms and stores of data.

(4)

(5)

C

ONTENTS Document Information ii Document History ii Acknowledgement ii Disclaimer ii Summary iii Contents v 1. Introduction ... 1 1.1 Fundamental approach... 1

1.2 Content and structure of this report ... 1

2. Guidance on data file types and storage... 2

2.1 Approach ... 2

2.2 Data Storage ... 2

2.3 Data Format ... 2

3. Guidance on using the NOKIS (Online) Metadata System... 3

3.1 Introduction ... 3 3.2 Description of website ... 3 3.2.1 Language ... 3 3.2.2 Login screen ... 3 3.2.3 Search screen... 4 3.2.4 Editor screen... 5

3.2.4.1 Metadata record information... 6

3.2.4.2 Identification information ... 6

3.2.4.3 Distribution information... 7

3.2.4.4 Data quality information ... 8

3.2.5 Templates screen... 8

3.2.6 Help ... 8

3.2.7 Keyword list ... 9

3.3 Known errors and improvements... 9

3.4 Frequently asked questions... 9

4. Further steps ... 9 Tables

None Figures

Fig. 1: Login screen 3

Fig. 2: Search screen 4

(6)

(7)

1. Introduction

With 36 partners undertaking 35 different tasks across the five year duration of FLOODsite, a considerable amount of data will be required and generated. The nature of this data will vary from task to task and may include survey data, images, video, drawings, reports, papers, modelling data etc. As an Integrated Project, many of the tasks rely upon other tasks and feed results through to later research. It will be important that team members are easily able to identify, access and use the various forms and stores of data.

1.1 Fundamental

approach

The primary objectives of FLOODsite do not include the creation and storage of data sets covering all areas of work. Hence, the function of any protocols for data management is primarily to facilitate team working, and hence achievement of FLOODsite deliverables, rather than the longer term store and wider provision of data.

Having considered the challenges posed by handling and exchanging such a diverse and large volume of data, and following the policy of KISMIH (keep it simple; make it happen) the following decisions were taken:

a) Partners would be asked to follow some basic rules regarding the use of common file types to minimise difficulties of shared access between partners

b) A central store of FLOODsite data would not be created; individual partners would be responsible for storing and referencing data sets that they accessed or created

c) Metadata, rather than actual data would be recorded, allowing partners to search and track data easily

d) The Metadata system should be of an international format such as described in ISO 19115. The system must provide a minimum description of the data available in FLOODsite and information on how to get it

These actions were implemented during 2005 and are now used by all FLOODsite team members.

1.2 Content and structure of this report

(8)

2

2. Guidance on data file types and storage

2.1 Approach

The exchange of data between Tasks within FLOODsite is essential for a fully integrated approach to the project. To facilitate this process, it is helpful to follow some basic rules as regards storage, file types and registration / description of the datasets (i.e. Metadata).

Registration of datasets (i.e. metadata) will be undertaken through the NOKIS project system (see Section 3). General guidance on data storage and file type is given below. Note that ‘data’ refers to all types of information such as reports, drawings, photos etc. as well as digital data sets such as topographic files, details of flows, water levels etc.

2.2 Data

Storage

The following principles should be followed by all partners:

• A partner collecting data should, wherever possible, store the data electronically

• This storage location must be persistent and secure for the duration of FLOODsite. A strategy for longer term data storage will also be developed during Year 3 of the FLOODsite project.

• A partner may choose to store the data at their site or via a third party if appropriate (e.g. in a managed data catalogue system).

• Storage need not be on-line.

• Data should be related to the tasks of the DoW / RIP.

• The owner of data is taken to be the providing partner from the project team unless specific owner / IPR information is provided.

2.3 Data

Format

All team members should make use of open formats for data, or widely use proprietary formats. Initial recommendations are to use the following formats:

• Image: JPEG, GIF, TIFF, EMF, WMF (standard formats) • Video: MPEG, AVI (no compression codecs)

• Text: plain ASCII or XML

• Numeric Data: deliminated ASCII (XYZ or grid) or XML • GIS Data:

Vector data:

SHP (but little accurate topology), ArcInfo Coverages (interchange format e00), ArcGIS geodatabase (MDB)

Raster data:

ArcInfo GRID (interchange format e00); ERDAS Image (IMG); MrSiD (high compressed image data) SID; GeoTIFF (TIF)

• Working Documents (text, numbers, images, tables): DOC or RTF or XLS • Final documents: PDF

Although not encouraged the following are allowed;

• Database: DBF (plus documentation on table relationships) • Binary formats: GRIB, NetCDF

(9)

3. Guidance on using the NOKIS (Online) Metadata System

3.1 Introduction

This section gives an overview of the current NOKIS website, describes how to use it and may serve as an introductory user manual. It contains a description of the website, explains key features and includes a section on frequently asked questions.

3.2 Description of website

The website is available under http://nokis.baw.de/floodsite. This is a temporary access for the moment which will possibly change over the duration of FLOODsite. Please note that the site only holds a description of all data (metadata) but not the data itself.

3.2.1 Language

The language of the website is set according to the language preferences of the internet browser used. If this needs to be changed one needs to go to the preferences page of the web browser, find the language preferences and change it accordingly. A reload of the website should then give the required language.

3.2.2 Login screen

The first page shows one tab with the login request as shown in Fig. 1. This step is required to limit access to FLOODsite team members only. Furthermore, depending on the type of login there are different access possibilities. In this section only the standard access by a normal user is explained in more detail. The standard user should login as follows:

! Login: user ! Password: user

(10)

4

3.2.3 Search screen

After having pressed ‘Login’ or whenever clicking on the ‘Search’ tab during a later stage of the web session the ‘Search’ page (Fig. 2) will appear.

Fig. 2: Search screen

The search page shows four tabs on top (‘Login’, ‘Search’, ‘Edit’, and ‘Templates’) which allows the user to switch in between the various screens of the website.

Without searching for any datasets the screen will display a list of all available datasets which are in the system (lower part of the screen). For each of these datasets the following information is shown:

! Record ID: this ID is provided by the system automatically

! Headline: describes the dataset and should be a carefully selected descriptor, it should contain

" the Task number of FLOODsite

" the location where the data comes from

" the type of data (discharge data, photo, video, report, etc.)

! Last Change: only for internal use. The date will be set automatically if the metadata set has been changed

! Choice: the datasets can be selected here. If one clicks on the box behind the data set it will be ticked. Selected metadata sets can either be deleted (see ‘Delete’ button on top of list) or exported in a ZIP format (see ‘ZIP Export’ button on top of list). One can also use the ‘Select all’ and ‘Deselect all’ buttons on top of the list to tick or untick all the boxes.

! Rights: shows the rights to work with the data (ED should be the standard and means that the dataset can be edited)

(11)

The list of records can be sorted using ‘Record no.’, ‘Headline’, and ‘Last Change’, either ascending or descending. The radio button left of each dataset can be used to select one dataset for editing. If the button is pressed and then the ‘Load’ button is clicked the system switches to the Editor screen and loads the selected dataset. In a similar way a copy of the dataset can be loaded into the Editor when pressing the ‘Load copy’ button.

On top of the search screen a line of text is shown where a free text search input can be given. The ‘Search for’ text is still in German and needs translation and so does the ‘Suchen’ (‘Search’) button on the right. However, this text search finds text in all input fields provided and is therefore a good tool to select special datasets.

3.2.4 Editor screen

The Editor page lets you edit the records and create new ones. Fig. 3 shows the screen for a given example of the measurements at Sylt island. The screen is split into two parts where the left one gives you some control buttons for the input and the right one shows you the actual data in different boxes. The headline on top left shows the record which is edited at the moment. This text cannot be changed or edited and neither can the date underneath the headline.

! ‘New document’: clicking on this button clears the form on the right and allows you to enter a new data set.

! ‘Save document’: this button saves the edited document to the database

There are four links (‘Metadata record information’, ‘Identification information’, ‘Distribution information’, and ‘Data quality information’) underneath the ‘Save document’ button which change the input boxes of the right side of this screen and lets you input different data (see below). Furthermore, the whole document can be saved as a template as well so that new records can be generated quickly when they are very similar to existing ones.

(12)

6

On the right side of the screen input forms for the data description are available. These are generally provided in boxes which have three to five buttons on top as follows:

! Submit: if any changes to an existing data set in the editor are made the ‘submit‘ button should be pressed for the system to accept the changes. One must subsequently also select ‘save document’ (left side of screen) to save the updated metadata set.

! Minimize: minimises the box so that only one headline remains for the whole box ! Maximize: opens all information in a box

! Remove: removes the contents of the box, please note that after removing the information you still have to press ‘Save document’ on the left side to actually save the new information you provided

! Add element: an additional element of the same type can be added for more input

All information marked by an asterisk is essential and should be provided. It is however always possible to save the documents even without this information.

3.2.4.1 Metadata record information

The input screen for this information is already given in Fig. 3. The key input here is the so-called ‘Headline’ which is shown in the list of search results on the ‘Search’ screen later. It is therefore essential that a good description of the data is provided here.

Further required information is about contact details. Once the user’s name is registered in the system the name can be found in the drop down list and all the other details are provided automatically.

3.2.4.2 Identification information

Fig. 4 shows the edit screen with the ‘identification information’ input boxes on the right side. These boxes contain information about the source of data and its temporal and spatial dimension.

(13)

A list of keywords is provided. Within FLOODsite a list of keywords was provided from which the user can select the most relevant ones for their data. This is regarded as very useful information to better search and find relevant datasets.

The ‘Abstract’ box contains a free text box which is used for giving further information of the data. A more or less detailed description can be given here. This is useful since the search is a free text search so that the search machine will find the information in this box as well.

The ‘Extent’ box needs special emphasis since it provides the input for the spatial and temporal extent of the data. This information should be given since some of the services NOKIS can provide are based on these data input.

The overall time to provide this data very much depends on the details of the metadata but may be estimated to be in the range of 5 minutes. To make use of the data details such as ‘Citation’ (so reports, papers, etc.), ‘Abstract’ (brief description of the data), and ‘Extent’ (both geographical and temporal extent) should be provided.

3.2.4.3 Distribution information

Fig. 5 shows the editor requesting information about distribution of the data. This information is provided by a link to an URL where more information can be received. There is only one box in this screen so that this box should open automatically when the link is requested (if not, press the ‘Add element’ or ‘Maximise’ button to show the input boxes available).

More elements (URLs) can be added by pressing the ‘Add element’ button. If no URLs for the data are available, the boxes can either be left empty or a remark as shown in Fig. 5 can be added. The estimated time to provide this data is in between 2 seconds (if no URL is available) and two minutes (if an URL for your data needs to be typed or copied), so will be very fast.

(14)

8

3.2.4.4 Data quality information

Fig. 6 shows the Editor screen with the data quality information. This is just one text box which can contain more detailed information on data quality, homogeneity of data sets and further information. There is only one input box under this link so that it should open automatically as well. Again, if this is not opened automatically, you will have to press the ‘Add Element’ or ‘Maximise’ button, depending on which one is shown.

Fig. 6: Edit screen showing the data quality information

It should be noted that a search for any data will go through all text boxes according to your search text provided. It would therefore be very useful, if information can be given here which other users might wish to find in the system. Keywords such as ‘gaps’, ‘interpolation’, extrapolation’, ‘resolution’, etc. might be useful here.

Time for data input will be in between 1-2 minutes depending on how detailed your report is. Please note that the more data are provided the easier it will be to find the relevant data set when searching for specific issues later.

3.2.5 Templates screen

The templates screen is empty at the moment since there are no templates saved yet. This can however be done from the edit page when a document is saved as a template. Templates can either be personal or overall templates where the latter can be used by any user of the website. Depending on how you logged in you will see the relevant information for you listed on this page. It is however proposed to not use the templates screen at the moment since it is unclear yet whether this will be deleted in the next version of the NOKIS website.

3.2.6 Help

(15)

might have some errors and bugs in it. An early observation was that the text was not translated automatically into the required language but is only in German. This service will be extended in the future.

3.2.7 Keyword list

FLOODsite has provided a keyword list (in English) which is able to describe (some of) the data sets. This list of keywords is regarded very important to find datasets which are very similar or belong to a similar group of data. It is expected that this list has to be improved in the near future.

3.3 Known errors and improvements

A list of observed errors and improvements should be given to the NOKIS team. They are able to provide changes very rapidly as soon as we are not requesting fundamental changes of the database layout. A first list of some observations can be found below:

! search screen: search text line and search button still in German when switched to English/other languages?

! search screen: an ‘empty’ search (just press ‘search’ with zero input in the search line) does not give you the full number of records, in our small example here the displayed line with the Sylt data set disappears

! search screen: the ‘XML’ option for the search results should be removed

! search screen: the print option (Ctrl-P) on the HMTL view page of the search results does not give the same output than the screen shows

! edit screen: the save button for the templates is still in German (‘Speichern‘)

! edit screen: in what cases does the user have to press “submit”, when “save”? This is not clear from intuition

! edit screen: under ‘identification information’ the box ‘Abstract’ should open automatically so that the user knows that this holds valuable information on the dataset

! edit screen: the topic categories should be adopted to FLOODsite (under ‘identification information’)

! edit screen: under ‘distribution information’ all boxes should open automatically ! edit screen: under ‘data quality information’ all boxes should open automatically ! overall: some help text is still in German

! overall: the background ‘language’ list should be reduced to some ‘European’ languages to make it more user-friendly (shorter)

3.4 Frequently asked questions

This section of the manual will be updated as soon as partners have started to work with the system and question came up in dealing with the system. Some of these faq might however be answered in looking to the known errors above.

4. Further steps

Partners in FLOODsite are now required to use the system intensively and input their data. The system will then profit from the feedback of the users and is expected to be improved over the coming months of FLOODsite.