Making Global Data Easier to Explore with AI and UN Data Commons

Making Global Data Easier to Explore with AI and UN Data Commons

Global statistics have a strange property: they are simultaneously abundant and hard to use. The numbers exist — population, emissions, trade, health, education — but they live in hundreds of separate portals, each with its own file formats, its own country naming conventions, and its own idea of what a "year" means. The gap between data availability and data usability is where most analysis projects quietly lose weeks.

A Google blog post describing work with the UN Data Commons platform frames the goal directly: making global data easier to explore. That is the subject of this article. Below, I separate what the source states from what is interpretation, and then focus on the part that is genuinely actionable today — the practical tooling you can set up to explore heterogeneous international data with AI assistance.

What the source states, and what it does not

The verified primary source is the Google AI Blog post Making global data easier to explore, published at https://blog.google/innovation-and-ai/technology/ai/google-un-data-commons-platform. The core verified statement is a goal, not a feature list: the effort is about making global data easier to explore, in the context of UN data work.

That is a deliberately narrow claim, and it is worth respecting. The post does not, on the evidence available here, specify query languages, SDK names, endpoints, model versions, pricing, or benchmark results. So this article will not invent them. Everything in the sections below about how to work with global data is general engineering practice for open-data work, not a description of a specific platform API.

Interpretation, clearly labelled: when an institution pairs a data commons with AI, the plausible value is not "the AI knows the answer." It is that AI can reduce the friction between a human question and a machine-readable query — matching a country name to a code, guessing which column means what, and drafting SQL against an unfamiliar schema.

Why global data resists exploration

Three obstacles appear in almost every cross-country project.

Identity mismatch. The same country appears as Kenya, KEN, KE, Republic of Kenya, and 肯尼亚 across different files. Joining on names fails silently.

Schema drift. One source stores year and value columns; another stores one column per year. One reports CO₂ in kilotonnes, another in million tonnes.

Metadata distance. The definition of an indicator lives in a PDF, while the numbers live in a CSV. Nothing connects them programmatically.

AI does not dissolve these problems. It shortens the loop: you can describe the join you want in plain language and get a draft, then verify it. The verification is still on you.

Requirements

Before installing anything, confirm you have:

• Python 3.10 or newer. Older versions break modern type hints and several data libraries.
• A package manager and a terminal. macOS, Linux, or WSL on Windows.
• Roughly 2 GB of disk space for a virtual environment and a modest local dataset.
• Optional: a Google Cloud project if you intend to query public datasets hosted in BigQuery.
• Optional: access to any LLM API you already use, if you want to try the natural-language-to-SQL pattern. The examples below treat this as a configurable endpoint, not a specific vendor.
• An open dataset to practise on. Any CSV or Parquet export from a national statistics office or an international agency works.

You do not need a GPU. This is a retrieval, harmonisation, and querying workflow, not a training workflow.

Step-by-step installation

1. Check your Python version

Run this to confirm the interpreter is new enough.

python3 --version

If it reports anything below 3.10, install a newer Python via your system package manager or pyenv before continuing.

2. Create an isolated virtual environment

Keeping dependencies out of your system Python prevents version conflicts later.

python3 -m venv ~/.venvs/globaldata
source ~/.venvs/globaldata/bin/activate

On Windows with WSL, the activation line is source ~/.venvs/globaldata/bin/activate as well; in PowerShell it would be ~/.venvs/globaldata/Scripts/Activate.ps1.

3. Upgrade the packaging toolchain

Fresh environments often ship with an outdated pip, which causes confusing resolver errors.

python -m pip install --upgrade pip setuptools wheel

4. Install the core data libraries

These cover HTTP retrieval, tabular manipulation, local analytical queries, and notebooks.

pip install pandas requests duckdb pyarrow python-dotenv jupyterlab

duckdb gives you SQL over local CSV and Parquet files without running a server — useful when a "database" is really just a folder of downloads.

5. Install the country-code helper

Country identity is the single most common source of broken joins, so install a maintained code table.

pip install pycountry

6. Optional: install the Google Cloud CLI

Do this only if you plan to query datasets hosted in BigQuery.

brew install --cask google-cloud-sdk

On Debian or Ubuntu, use the official apt repository instead of brew.

7. Optional: authenticate to Google Cloud

This opens a browser window and stores application default credentials locally.

gcloud auth application-default login

8. Optional: select a project and enable the API

Set the project you want billed for any queries you run.

gcloud config set project YOUR_PROJECT_ID
gcloud services enable bigquery.googleapis.com

9. Optional: install the Python client

This is the library that lets scripts call BigQuery directly.

pip install google-cloud-bigquery db-dtypes

10. Verify the environment

Run this to confirm the essential imports resolve.

python -c "import pandas, duckdb, pycountry, requests; print('environment ready')"

Configuration

Keep credentials and endpoints out of your source files.

1. Create a configuration file

This writes a private env file in your home directory. Replace every placeholder with your own values.

cat > ~/.globaldata.env <<'EOF'
DATA_API_ENDPOINT=https://your-data-portal.example/api/v1
DATA_API_KEY=replace-with-your-key
MODEL_ENDPOINT=https://your-llm-provider.example/v1
MODEL_API_KEY=replace-with-your-model-key
MODEL_NAME=replace-with-your-model-id
GCP_PROJECT=your-project-id
EOF

2. Restrict permissions on the file

Credentials in a world-readable file are a common and avoidable mistake.

chmod 600 ~/.globaldata.env

3. Load the configuration in Python

This snippet reads the file at runtime without hardcoding secrets.

import os
from dotenv import load_dotenv

load_dotenv(os.path.expanduser("~/.globaldata.env"))

DATA_API_ENDPOINT = os.environ["DATA_API_ENDPOINT"]
GCP_PROJECT = os.environ.get("GCP_PROJECT")

The DATA_API_ENDPOINT placeholder is deliberate. Any portal you use will have its own documented base URL; the patterns below are agnostic to which one that is.

Usage examples

Example 1: Normalise country identifiers

This is the highest-leverage twenty lines of code in any cross-country analysis.

import pycountry

ALIASES = {
    "Republic of Korea": "KOR",
    "Korea, Rep.": "KOR",
    "Viet Nam": "VNM",
    "Russian Federation": "RUS",
    "Côte d'Ivoire": "CIV",
}

def to_iso3(name: str) -> str | None:
    name = name.strip()
    if name in ALIASES:
        return ALIASES[name]
    try:
        return pycountry.countries.lookup(name).alpha_3
    except LookupError:
        return None

print(to_iso3("Kenya"), to_iso3("Korea, Rep."), to_iso3("Atlantis"))

The last call returning None is the point: unmapped values should be visible, not silently dropped. Log them and extend ALIASES rather than guessing.

Example 2: Load heterogeneous files into one queryable table

DuckDB can read CSV and Parquet directly and union them, which avoids an import step.

import duckdb

con = duckdb.connect("globaldata.duckdb")

con.execute("""
    CREATE OR REPLACE TABLE observations AS
    SELECT
        indicator_code,
        UPPER(country_iso3) AS country_iso3,
        CAST(year AS INTEGER) AS year,
        CAST(value AS DOUBLE) AS value,
        source_file
    FROM read_csv_auto('downloads/*.csv', filename = true)
    WHERE value IS NOT NULL
""")

print(con.execute("""
    SELECT country_iso3, COUNT(*) AS rows
    FROM observations
    GROUP BY 1 ORDER BY 2 DESC LIMIT 10
""").fetchdf())

The filename = true option preserves provenance, so every row can be traced back to the file it came from.

Example 3: Draft a query from a plain-language question

This pattern sends your schema plus a question to a language model and asks for SQL back. It is a drafting aid, not an oracle.

import os, requests

SYSTEM = """You translate questions into DuckDB SQL over this schema:

observations(indicator_code VARCHAR, country_iso3 VARCHAR, year INTEGER, value DOUBLE)

Rules:
- Return only the SQL statement.
- Use read-only SELECT statements.
- Never emit INSERT, UPDATE, DELETE, DROP, or ATTACH.
"""

def draft_sql(question: str) -> str:
    response = requests.post(
        f"{os.environ['MODEL_ENDPOINT']}/chat/completions",
        headers={"Authorization": f"Bearer {os.environ['MODEL_API_KEY']}"},
        json={
            "model": os.environ["MODEL_NAME"],
            "temperature": 0,
            "messages": [
                {"role": "system", "content": SYSTEM},
                {"role": "user", "content": question},
            ],
        },
        timeout=60,
    )
    response.raise_for_status()
    return response.json()["choices"][0]["message"]["content"].strip()

print(draft_sql("Which five countries had the highest value in 2020?"))

Example 4: Execute the draft against a read-only connection

Opening the database read-only is a cheap, effective guardrail.

import duckdb

con = duckdb.connect("globaldata.duckdb", read_only=True)

sql = draft_sql("Which five countries had the highest value in 2020?")
print(sql)

if sql.upper().lstrip().startswith("SELECT"):
    print(con.execute(sql).fetchdf())
else:
    print("Refused: statement is not a SELECT.")

The startswith("SELECT") check is crude and easily bypassed by a determined prompt. Treat it as a tripwire, not a security boundary. For real deployments, restrict the database user's privileges at the engine level.

Example 5: Attach an indicator definition to its numbers

Metadata is what makes a number interpretable. Keep a small lookup table alongside the data.

con = duckdb.connect("globaldata.duckdb")

con.execute("""
    CREATE OR REPLACE TABLE indicators AS
    SELECT * FROM (VALUES
        ('EN.GHG.CO2.MT.CE.AR5', 'Carbon dioxide emissions', 'million tonnes'),
        ('SP.POP.TOTL',           'Total population',         'people')
    ) AS t(indicator_code, label, unit)
""")

print(con.execute("""
    SELECT i.label, i.unit, o.country_iso3, o.year, o.value
    FROM observations o
    JOIN indicators i USING (indicator_code)
    WHERE o.country_iso3 = 'KEN'
    ORDER BY o.year DESC
    LIMIT 5
""").fetchdf())

Now a retrieved figure arrives with its unit attached, which prevents a whole class of downstream errors.

Pitfalls and open limits

AI drafts are not citations. A generated SQL statement can reference a column that does not exist or filter on a country code that was never in the data. Always run it against a small slice first.

Language models do not know your indicator. They know that SP.POP.TOTL looks like a World Bank code, but they cannot confirm what a specific portal means by it. The definition must come from the publisher.

Provisioning is invisible until it is not. This article's installation steps describe general-purpose open-data tooling. Any specific platform built around the UN Data Commons effort will have its own onboarding, terms, and rate limits, which the available source does not describe. Read those documents before you build dependencies on them.

Harmonisation is never finished. New country names, new indicator revisions, and new vintages appear constantly. Budget for a maintenance loop, not a one-time cleanup.

Aggregation hides disagreement. When two agencies report different values for the same indicator, a join will happily produce both. Decide explicitly whether to prefer one source, average them, or surface the disagreement.

Conclusion

The promise of AI in global data work is narrower and more useful than it first sounds. It is not that the model knows the statistics. It is that the model can sit between a human question and a fragmented storage layer, drafting the join, proposing the code mapping, and cutting the time from "I wonder" to "here is a query I can check."

The UN Data Commons effort, as described in the source, targets exactly that friction: making global data easier to explore. The practical response is to build a small, reproducible environment around whatever data you already have — a virtual environment, DuckDB for local SQL, pycountry for identity, and a configurable model endpoint for drafting. Three rules keep it honest: normalise identifiers rather than names, keep metadata next to the numbers, and never execute a generated query you have not read.

Start with one dataset and one question. The tooling takes an afternoon; the habit of verifying what the model produced is what makes the results trustworthy.