One analysis-ready dataset combining weather, food prices, energy and economic data for 5 countries, so an analyst can ask a cross-country question without doing the joins by hand.
Built with a 4-person team for Data Engineering Principles & Practice at Johns Hopkins (Spring 2026). I built the infrastructure (Docker setup, scheduled refresh, Flask API) and wrote the product spec.
Weather, food prices, energy and economic data are all public. But they come from different sources in different shapes: some monthly, some yearly, some keyed by country name, some by code. An analyst who wants to ask "did a dry year line up with higher rice prices and slower growth?" spends most of their time cleaning and joining files before they can answer it.
One analysis-ready table with one row per country per month, served through an API and refreshed on its own. The analyst starts at the question, not the joins.
| Decision | Why |
|---|---|
| Country code + month as the key | Different sources name countries differently. The ISO3 country code plus year-month gives every source one join key and one time scale. |
| Keep gaps visible | Outer joins keep every record. Missing coverage (for example, no US food-price data) stays as a blank, so a gap in the source never looks like a processing error. |
| Spread yearly data across months | Energy and economic data are yearly. Spreading them evenly across months lets them sit next to monthly weather and food data without inventing seasonal patterns. |
| 5 countries, 2022 onward | US and Brazil give strong economic and energy coverage; India, the Philippines and Nigeria give food-price coverage and different climate exposure. Starting in 2022 keeps the overlap useful. |
| Weekly refresh, not real-time | The sources update slowly. A weekly scheduled job is cheaper and simpler, and plenty for this kind of analysis. |
docker compose up starts the database, loads the full history on first start, then serves the API.I wrote a product spec on top of the pipeline: user types, value proposition, success targets and a three-phase roadmap.
These are targets set in the spec, not measured results.