Archives FAQs

| January 14, 2021

You can share your data on HDX under one of the following Creative Commons licenses

Creative Commons Attribution for Intergovernmental Organisations (CC BY-IGO)

[ more information: deed | license]

Under the CC BY-IGO license, you are free to share (copy and redistribute the material in any medium or format) and or adapt (remix, transform, and build upon the material) for any purpose, even commercially. The licensor cannot revoke these freedoms as long as you follow the license terms. The license terms are that you must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. Additionally, you may not apply legal terms or technological measures that legally restrict others from doing anything the license permits. When the Licensor is an intergovernmental organization, disputes will be resolved by mediation and arbitration unless otherwise agreed.

Creative Commons Attribution International(CC BY)

[ more information: deed | license]

Under the CC BY license, you are free to share (copy and redistribute the material in any medium or format) and or adapt (remix, transform, and build upon the material) for any purpose, even commercially. The licensor cannot revoke these freedoms as long as you follow the license terms. The license terms are that you must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. Additionally, you may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.

Creative Commons Attribution-ShareAlike (CC BY-SA)

[ more information: deed | license]

Under the CC BY-SA license, you are free to share (copy and redistribute the material in any medium or format) and or adapt (remix, transform, and build upon the material) for any purpose, even commercially. The licensor cannot revoke these freedoms as long as you follow the license terms. The license terms are that you must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original. Additionally, you may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.

Open Database License (ODC-ODbL)

[ more information: deed | license]
Under the ODC-ODbL license, you are free:

  • To Share: To copy, distribute and use the database.
  • To Create: To produce works from the database.
  • To Adapt: To modify, transform and build upon the database.

As long as you:

  • Attribute: You must attribute any public use of the database, or works produced from the database, in the manner specified in the ODbL. For any use or redistribution of the database, or works produced from it, you must make clear to others the license of the database and keep intact any notices on the original database.
  • Share-Alike: If you publicly use any adapted version of this database, or works produced from an adapted database, you must also offer that adapted database under the ODbL.
  • Keep open: If you redistribute the database, or an adapted version of it, then you may use technological measures that restrict the work (such as DRM) as long as you also redistribute a version without such measures.

Open Data Commons Attribution License (ODC-BY)

[ more information: deed | license]
Under the ODC-BY license, you are free:

  • To Share: To copy, distribute and use the database.
  • To Create: To produce works from the database.
  • To Adapt: To modify, transform and build upon the database.

As long as you:

  • Attribute: You must attribute any public use of the database, or works produced from the database, in the manner specified in the license. For any use or redistribution of the database, or works produced from it, you must make clear to others the license of the database and keep intact any notices on the original database.

Open Data Commons Public Domain Dedication and License (PDDL)

[ more information: deed | license]
Under the ODC-PDDL license, You are free:

  • To Share: To copy, distribute and use the database.
  • To Create: To produce works from the database.
  • To Adapt: To modify, transform and build upon the database.

As long as you:

  • Blank: This section is intentionally left blank. The PDDL imposes no restrictions on your use of the PDDL licensed database.

Public Domain/No restrictions (CC0)

[ more information: deed | license]

Under the terms of this license you are free to use the material for any purpose without any restrictions.

Multiple Licenses

The dataset contains data having different licenses or terms of use. The details of these licenses or terms of use should be listed in the file.

Other

Any other license or terms of use which are listed in the description of the dataset, or in the metadata fields of the dataset, or any other place in the dataset such as a specific license or terms of use file that is included as part of the dataset files.

Note: If you would like us to include other license options for data shared through HDX, please send us an email with your request at hdx@un.org.


| January 14, 2021

HDX TERMS OF SERVICE

[ March 2021]

The Humanitarian Data Exchange (HDX) is an open data platform managed by the United Nations Office for the Coordination of Humanitarian Affairs (OCHA) through its Centre for Humanitarian Data (the Centre). These Terms of Service (hereafter 'Terms' or 'these Terms') describe how HDX is managed and how the platform should be used. OCHA will update these Terms as needed, and will post notice of significant updates on HDX and through the HDX mailing list. 

HDX organizations and users are bound by these Terms. If you do not agree with the Terms, you should discontinue use of HDX. If you have any questions or comments about these Terms or HDX, please visit our Frequently Asked Questions or send an email to hdx@un.org

-

ACCOUNT MANAGEMENT
  1. User account. HDX is an open platform and anyone can use it without creating a user account. Signing up with HDX gives users access to additional features such as the ability to receive notifications about data; joining an organization as a member, editor or admin; and requesting access to datasets shared via HDX Connect, among other benefits.
  2. Organization account. Data can only be shared on HDX by approved organizations. Organizations can represent a formal legal entity such as a non-governmental organization, or an informal collective such as an Information Management Working Group. OCHA reviews requests to create an organization account to: (1) verify the identity of the requester and (2) determine whether the data that will be shared meets the requirements set out in the DATA SCOPE AND CRITERIA section below.
  3. You may delete your user or organization account at any time. When you delete your account, OCHA will delete any personal data we collected in order to create the account. When an organization account is deleted, the data shared by the organization is also deleted from HDX. 

-

DATA SCOPE AND CRITERIA
  1. There are three categories of humanitarian data which may be shared on HDX:
    a. Data about the context in which a humanitarian crisis is occurring (e.g. administrative boundaries, locations of schools, health facilities and other physical infrastructure, and baseline socio-economic indicators).
    b. Data about the people affected by the crisis and their needs (e.g. needs assessment data, movement data and locations of affected people).
    c. Data about the response by organizations seeking to help those who need assistance (e.g. who-is-doing-what-where, community perception surveys, and funding levels).
  2. All data shared on HDX must meet the following criteria:
    a.
    Public and private datasets may not contain any personal data. Aid worker contact details may be shared within a private dataset, if those aid workers have provided consent. Personal data is information, in any form, that relates to an identified or identifiable natural person.
    b. Public and private datasets may not contain any sensitive non-personal data. This includes information which, while not relating to an identified or identifiable natural person, may, by reason of its sensitive context, put certain individuals or groups of individuals at risk of harm.
    c. Data must have been collected in a fair and legitimate manner with a defined purpose and in line with principles of necessity and proportionality.
    d. Data must be shared in a supported data format. HDX supports all common data formats and offers built-in preview support for CSV, TXT, XLS, and JSON formats. Map previews are possible from geographic data in zipped shapefile, KML and GeoJSON formats.
  3. Organizations should keep their data on HDX up-to-date in order to present the latest available information.

-

SHARING DATA
  1. There are three ways to share data on HDX:
    a. Public: Data is accessible to anyone who visits HDX, whether or not they are a registered user.
    b. Private: Data is accessible only to registered users who are members of the organization that uploaded the data on HDX.
    c. HDX Connect: The metadata of a dataset is available and the contributing organization can decide whether or not to grant access to the full dataset when requested by a registered user.
  2. Organizations must specify an appropriate license for all data they share publicly. Organizations are free to choose the license for their data. We have suggested some options here.
  3. Organizations may use HDX to share data from other sources if the applicable license allows for onward sharing.
  4. After downloading a public dataset, users must follow the applicable license when using and sharing the data.
  5. Organizations may use the HDX Connect feature to direct users to data hosted outside of HDX. In such cases, organizations should link directly to the specific dataset described on HDX and not to a more general landing page of an external platform.
  6. When an organization grants access to data requested via HDX Connect, the data does not pass through the HDX infrastructure.

 

-

DATA REVIEW
  1. In order to ensure data quality and to prevent any sensitive data from being exposed through HDX, OCHA reviews all datasets that are shared publicly or privately on the platform. This review consists of:
    a.
    An automated scan for sensitive data using Google’s Data Loss Prevention (DLP) tool, to flag and prioritize data for manual review by OCHA.
    b. A manual review based on a quality assurance checklist that includes the completeness of metadata, the relevance of the data to humanitarian action, the integrity of the data resources, and the absence of any sensitive data, among other criteria.
  2. If the manual review under 13(b) shows that a dataset contains personal or sensitive data, the dataset is placed ‘under review’. While data is under review, users will only be able to consult the metadata.
  3. For microdata such as household survey results, OCHA runs a disclosure risk assessment to assess the risk of a person or group being re-identified. All datasets labeled as ‘microdata’ by the contributing organization at the point of upload are automatically placed under review. The dataset will remain under review until OCHA is able to determine that the risk of re-identification is below the risk threshold and that any sensitive data has been removed from the dataset by the organization. More information about this process is available here.
  4. If a user notices personal or sensitive data shared through the HDX platform they should contact hdx@un.org immediately to request that the data be removed.

-

DATA MANAGEMENT
  1. HDX is built using CKAN, an open-source data management system.
  2. Data that is uploaded to HDX is stored by OCHA on servers provided by Amazon Web Services. Data is encrypted in transit and at rest. The servers are located in Virginia, the United States of America.
  3. All data uploaded to HDX is sent via Google’s DLP API for automated scanning for sensitive data using the DLP algorithm. Data is encrypted in transit and scanned through DLP’s content method on servers located in the European Union. Data is not retained by Google in this process.
  4. OCHA will never alter the values within datasets shared through HDX without prior permission from the contributing organization.
  5. Data shared privately through the HDX platform will never be shared further by OCHA without prior permission from the contributing organization.
  6. OCHA will make a dataset private if it is found to violate these Terms and will contact the contributing organization to discuss next steps.
  7. Deleted datasets cannot be retrieved by users, but will continue to exist in backups of the HDX database which are maintained for 30 days.

-

GENERAL DISCLAIMER OF LIABILITY
  1. Organizations are responsible for the data they share on HDX. OCHA assumes no liability whatsoever for data shared on HDX. While OCHA upholds a high standard for the quality and timeliness of the data shared on HDX, we cannot verify data accuracy. Sharing data through HDX does not imply the transfer of any rights over this data to OCHA. OCHA disclaims all warranties, whether express or implied.
  2. Data and information on HDX do not imply the expression or endorsement of any opinion on the part of OCHA or the United Nations. This includes opinions concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries.

-

PRIVACY NOTICE
  1. User contact details are only shared with the administrator of an HDX organization if the user requests access to an HDX Connect dataset.
  2. OCHA upholds the highest standard of data protection for the personal data of HDX users and organization administrators. In case such personal data is exposed, OCHA will notify all affected individuals and remedy the incident.
  3. OCHA continually seeks to understand the behavior of users on the HDX platform in order to make improvements. To do so, OCHA uses third-party analytics services, including Google Analytics and Mixpanel. Both of these services use cookies stored on users' devices to send encrypted information to Google Analytics and Mixpanel about how users arrived at HDX, what pages they visited on HDX, and their actions within those pages. Similar tracking is performed when users access HDX via our API or when directly downloading files from a shared link. OCHA does not send identifying information (including names, usernames, or email addresses) to either Google Analytics or Mixpanel. Google Analytics' and Mixpanel's use of the data collected from the HDX platform is governed by their respective Terms of Use.
  4. If you would like to disable the tracking described above under clause 28, you can install the Google Analytics Opt-out Browser Add-on to disable Google Analytics tracking. Mixpanel respects "Do Not Track" settings in web browsers. Follow the instructions in this guide to prevent your browser from sending data to Mixpanel. The data collected by these tracking systems will be retained indefinitely in order to understand how user behavior is changing over time.
  5. Emails sent by OCHA to registered HDX users may contain web beacons, which allow OCHA to track information about how many people have viewed its email campaigns. OCHA will never share personal data from this tracking with third parties other than with MailChimp, our mailing list provider, which has access by default. The data collected by this tracking system will be retained indefinitely in order to understand how readership of the emails is changing over time.

-

APPLICABLE GUIDANCE AND POLICY
  1. OCHA is mandated by United Nations General Assembly Resolution 46/182 and guided by the Humanitarian Principles. OCHA is governed by the applicable guidance and policies established by the United Nations General Assembly and the United Nations Secretariat. Notably, personal data is processed according to the 1990 Guidelines for the Regulation of Computerized Data Files and in line with the UN Principles on Personal Data Protection and Privacy.

 


| January 11, 2021

If you have any questions about these resources, we will do our best to answer them. We would also love to hear about how you are using them for your work.

Please contact us at: hdx@un.org. Sign up to receive our newsletter here.


| January 11, 2021

All kinds of utility functions have been coded over time for use internally, so since we think these have value externally, it was decided that they should be packaged into the HDX Python Utilities library which has library API-level documentation available online.


| January 11, 2021

Humanitarian projects frequently require handling countries, locations and regions in particular dealing with inconsistent country naming between different data sources and different coding standards like ISO3 and M49. The HDX Python Country library was created to fulfill these requirements and is a dependency of the HDX Python API. It is also very useful as a standalone library and has library API-level documentationavailable online.


| January 11, 2021

The HXL Proxy is a tool for validating, cleaning, transforming, and visualising HXL-tagged data. You supply an input url pointing to a tabular or JSON dataset and then create a recipe that contains a series of steps for transforming the data. The result is a download link that you can share and use in HDX, and the output will update automatically whenever the source dataset changes. Full user documentation is available in the HXL Proxy wiki.
The HXL Proxy is primarily a web wrapper around the libhxl-python library (see above), and makes the same functionality available via RESTful web calls.


| January 11, 2021

HDX provides a suite of tools that leverage HXLated datasets:

  1. QuickCharts automatically generates embeddable, live data charts, graphs and key figures from your data. It uses the HXL hashtags to guess the best charts to display, but you can then go in and override with your own preferences.
  2. HXL Tag Assist allows you to find hashtag examples and definitions, and see how data managers are using the hashtags in their data.
  3. Data Check provides help with data cleaning for humanitarian data, automatically detecting and highlighting common errors. It includes validation against CODs and other vocabularies.

| January 11, 2021

Third party support for R is available via the package rhxl. It has basic support for reading HXLated files to make them available for advanced data-processing and analytics inside R.


| January 11, 2021

libhxl-js is a library for HXL written in Javascript. It supports high-level filtering and aggregation operations on HXL datasets. Its programming idiom is similar to libhxl-python, but it is smaller and contains fewer filters and no data-validation support.


| January 11, 2021

The most well developed HXL library, libhxl-python, is written in Python. The most recent versions support Python 3 only, but there are earlier versions with Python 2.7 support. Features of the library include filtering, validation and the ingestion and generation of various formats. libhxl-python uses an idiom that is familiar from JQuery and other Javascript libraries; for example, to load a dataset, you would use simply

import hxl 
source = hxl.data('http://example.org/dataset.xlsx')

As in JQuery, you process the dataset by adding additional steps to the chain. The following example selects every row with the organisation "UNICEF" and removes the column with email addresses:

source.with_rows('#org=UNICEF').without_columns('#contact+email')

The library also includes a set of command-line tools for processing HXL data in shell scripts. For example, the following will perform the same operation shown above, without the need to write Python code:

$ cat dataset.xlsx | hxlselect -q "#org=UNICEF" | hxlcut -x '#contact+email'

There is library API-level documentation available online.