pathlib, JSON/YAML config and environment variables

The problem we are solving

A support-bot project has a small script, scripts/show_config.py, that reads config/app.json and shows the model name and temperature. Its author always stands in the project root and runs python scripts/show_config.py, and it works. One day a colleague stepped into the scripts/ folder, ran the same script, and got a FileNotFoundError — with the file exactly where it has always been.

Two more things happened that week. The server had APP_DEBUG=false set, yet the logs showed debug mode switched on. And someone wrote temperature: 0 in the config so that the model would give the same answer every time — but the program ran with 0.7. None of these raised an error.

None of the three is a problem with the code's logic. The problem is how the program reads the outside world — the file system, config files, environment variables. In this chapter we make that reading dependable: paths with pathlib, nested config with JSON and YAML, and secrets with .env — each with validation, so that a bad config stops at the start instead of quietly running.

By the end of this chapter you can

• Build paths that work on any operating system with pathlib.Path, using / instead of joining os.path strings
• Find files from the script's own location, independent of where the terminal is standing
• Read nested JSON and YAML config, and recognise the ways yaml.safe_load quietly changes types
• Read values from environment variables and a .env file, convert them to the right type and validate them
• Write a config loader that stops with a clear message when a value is wrong or missing

Prerequisites: pyproject.toml, lockfiles and the src/ layout — especially uv add and uv run.

pathlib: a path is an object, not a string

In older Python code a path was a string, glued together with + "/" + or os.path.join. The Path class from the pathlib module treats a path as an object whose parts you can ask for:

from pathlib import Path

config_file = Path("config") / "models" / "chat.yaml"

print(config_file.as_posix())
print(config_file.name)
print(config_file.stem)
print(config_file.suffix)
print(config_file.parent.as_posix())
print(config_file.with_suffix(".json").as_posix())
print(config_file.parts)

config/models/chat.yaml
chat.yaml
chat
.yaml
config/models
config/models/chat.json
('config', 'models', 'chat.yaml')

Here / is not division — used with a Path, it joins path parts and picks the right separator for the operating system. The output is printed with .as_posix() so it looks the same on every machine; a plain print(config_file) shows config\models\chat.yaml on Windows and config/models/chat.yaml on macOS and Linux.

Gluing strings makes the difference visible:

from pathlib import Path

base = "config"
print(base + "/" + "app.json")
print(Path(base) / "app.json")
print(repr(Path(base) / "app.json"))

On Windows this prints:

config/app.json
config\app.json
WindowsPath('config/app.json')

The first line is a hand-built string. Windows usually accepts /, but once you pass such paths to other tools or compare them as strings, two kinds of separator mixed together cause confusion. Path builds each path by its own system's rules. On macOS or Linux the second line shows config/app.json and the third PosixPath('config/app.json').

/ only works when at least one side is a Path. In "config" / "app.json" both sides are strings, so Python reads it as division and raises a TypeError. Make the leftmost part a Path and the rest can stay strings.

A few more methods for everyday work, which the examples below use:

• path.exists(), path.is_file(), path.is_dir() — whether the path is there, and what kind of thing it is
• path.read_text(encoding="utf-8") and path.write_text(text, encoding="utf-8") — read or write a whole file without open
• path.mkdir(parents=True, exist_ok=True) — make a folder, including the ones in between, without an error if it exists
• folder.glob("*.txt") — every file in a folder matching a pattern; rglob searches subfolders too
• path.resolve() — turn a relative path into an absolute one

Write encoding="utf-8" every time. On Windows, Python's default encoding is often not UTF-8, so a prompt file containing Bengali text or an emoji reads fine on a Mac and gives garbled characters or a UnicodeDecodeError on Windows.

Relative to what?

The script from the start of the chapter looks like this:

from pathlib import Path

config_path = Path("config/app.json")
print(config_path.read_text(encoding="utf-8"))

From the project root, python scripts/show_config.py prints the config. But step into scripts/ and run python show_config.py:

Traceback (most recent call last):
  File "show_config.py", line 4, in <module>
    print(config_path.read_text(encoding="utf-8"))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Lib\pathlib.py", line 1028, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...\Lib\pathlib.py", line 1014, in open
    return io.open(self, mode, buffering, encoding, errors, newline)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'config\\app.json'

(On your machine the traceback shows full paths; they are shortened here.) A relative path is always looked up from the current working directory — wherever the terminal is standing — not from where the script file lives. There is no config/app.json inside scripts/, so the file was not found.

In production this happens even more: you do not always know which folder a cron job, a service manager or a container will start your program from. The fix is to tie the path to the script's own location. Inside every module, __file__ holds that file's path:

from pathlib import Path

PROJECT_ROOT = Path(__file__).resolve().parent.parent
config_path = PROJECT_ROOT / "config" / "app.json"
print(config_path.read_text(encoding="utf-8"))

{"model": "chat-small", "temperature": 0.2}

Now it prints the same line whether you run it from the project root, from scripts/, or from somewhere else on the disk entirely with a full path. Step by step: Path(__file__) is show_config.py; .resolve() makes it absolute; the first .parent is scripts/; the second .parent is the project root.

Do not drop .resolve(). In some situations __file__ is a relative path, and then .parent.parent depends on the current folder again — the very problem you were trying to avoid.

JSON: nested config and the zero trap

Config in AI projects is almost always nested — app settings, model settings and retrieval settings in separate groups. json.loads turns a JSON string into nested dictionaries (from a file, json.loads(path.read_text(encoding="utf-8"))):

import json

raw = """
{
  "app": {"name": "support-bot", "debug": false},
  "llm": {"model": "chat-small", "temperature": 0, "max_tokens": 512},
  "retrieval": {"top_k": 4}
}
"""
config = json.loads(raw)

print(config["llm"]["model"])
print(config["retrieval"]["top_k"])
print(type(config["app"]["debug"]))

temperature = config["llm"].get("temperature") or 0.7
print(temperature)

temperature = config["llm"].get("temperature", 0.7)
print(temperature)

chat-small
4
<class 'bool'>
0.7
0

JSON's false becomes Python's False and numbers stay numbers — JSON keeps types. But look at the last two lines. The config plainly says "temperature": 0, yet the first approach gave 0.7.

x or default means "x if x is truthy, otherwise default". And in Python, 0, 0.0, "", [] and {} are all falsy. So or cannot tell "no value" from "a value, and it is zero". A temperature of 0 makes the model give its most likely answer every time, which is what you want for tests and evaluation; or quietly turns it into 0.7, and all you see is answers that differ from run to run.

Use .get(key, default) for defaults — it only uses the default when the key is truly absent. or is right only when you genuinely want to throw away zero and empty values too.

YAML: easy for people, but it guesses types

Many teams write config in YAML, because it allows comments and has fewer brackets and quotes. Python does not come with a YAML reader; add one to the project:

uv add pyyaml

The package is called pyyaml, but you import it as import yaml. Now read a config:

import yaml

raw = """
llm:
  model: chat-small
  temperature: 0.2
features:
  streaming: yes
  cache: off
python_version: 3.10
api_version: "3.10"
"""
config = yaml.safe_load(raw)
print(config["llm"])
print(config["features"])
print(config["python_version"], type(config["python_version"]).__name__)
print(config["api_version"], type(config["api_version"]).__name__)

{'model': 'chat-small', 'temperature': 0.2}
{'streaming': True, 'cache': False}
3.1 float
3.10 str

Unlike JSON, YAML does not make you spell out types — the parser guesses. Most of the time the guess is right, but in two places it is quietly wrong. yes, no, on and off become booleans — the file said cache: off and you got False. And 3.10 is read as a decimal number whose value is 3.1; as a version number that is a different thing entirely. The quoted "3.10" stays a string.

The rule is simple: any value that looks like a number or a yes/no but is really text — version numbers, IDs, postal codes — goes in quotes in YAML.

And always use yaml.safe_load. The yaml.load(text) you see in old tutorials now fails outright with TypeError: load() missing 1 required positional argument: 'Loader'. That change is deliberate: some loaders can build arbitrary Python objects from instructions inside the YAML file, so reading an untrusted YAML file means running someone else's code. safe_load only builds plain types — dictionaries, lists, strings, numbers, booleans, None.

Environment variables: everything is a string

API keys, database addresses, which model to use — values that change from machine to machine or have to stay secret live in environment variables, not in code or in committed config. Python reads them with os.environ or os.getenv. For the example, two variables are set inside the code itself; in real work they come from the shell, the server or the container:

import os

os.environ["APP_DEBUG"] = "false"
os.environ["APP_PORT"] = "8000"

debug = os.getenv("APP_DEBUG")
print(repr(debug), bool(debug))

if debug:
    print("debug mode is ON")

port = os.getenv("APP_PORT")
print(port + "1")
print(int(port) + 1)

print(os.getenv("LLM_API_KEY"))
print(os.getenv("LLM_TIMEOUT", "30"))

'false' True
debug mode is ON
80001
8001
None
30

Environment variables are always strings. "false" is a non-empty five-character string, so if debug: treats it as true — that is exactly how the server at the start of the chapter ran in debug mode with "false" set. In the same way, "8000" + "1" is concatenation, not addition. No error, just a program doing the wrong thing.

So convert every environment variable to the right type as soon as you read it. For numbers, int() or float() is enough — they raise ValueError on bad input themselves. For booleans Python has no built-in conversion (bool("false") is True), so you write your own:

def parse_bool(value: str) -> bool:
    normalized = value.strip().lower()
    if normalized in {"1", "true", "yes", "on"}:
        return True
    if normalized in {"0", "false", "no", "off"}:
        return False
    raise ValueError(f"not a boolean: {value!r}")


for raw in ["false", "True", " yes ", "0"]:
    print(repr(raw), "->", parse_bool(raw))

parse_bool("flase")

'false' -> False
'True' -> True
' yes ' -> True
'0' -> False
Traceback (most recent call last):
  File "main.py", line 13, in <module>
    parse_bool("flase")
  File "main.py", line 7, in parse_bool
    raise ValueError(f"not a boolean: {value!r}")
ValueError: not a boolean: 'flase'

The last line is the whole value of this function. A misspelt "flase" does not quietly become True or False — the program stops at the start and says which value it could not understand.

Keep the difference between os.environ["NAME"] and os.getenv("NAME") in mind too. The first raises KeyError: 'LLM_API_KEY' when the variable is missing; the second quietly returns None, which may come back a few minutes later as a 401 Unauthorized from an API call — by which point the cause is much harder to find. Check any value the program cannot run without at the very start.

The .env file: an environment for your own machine

Setting ten variables every time you open a terminal is not realistic. The usual answer is a .env file in the project root and the python-dotenv package, which reads that file and puts the values into os.environ:

uv add python-dotenv

Again the package name and the import name differ: the package is python-dotenv, the import is dotenv. A .env file:

# Local secrets. Never commit this file.
LLM_API_KEY=sk-test-placeholder-1234
LLM_MODEL=chat-small
APP_DEBUG=true

(sk-test-placeholder-1234 is a made-up value; never put a real key in any document, screenshot or commit.) The code to read it, again building the path from the script's location:

import os
from pathlib import Path

from dotenv import load_dotenv

env_file = Path(__file__).resolve().parent / ".env"
found = load_dotenv(env_file)

print(found)
print(os.getenv("LLM_MODEL"))
print(os.getenv("APP_DEBUG"))
print(os.getenv("LLM_API_KEY")[:7] + "...")

uv run main.py:

True
chat-small
true
sk-test...

Now the same program, but with LLM_MODEL already set in the shell (macOS/Linux or Git Bash):

LLM_MODEL=chat-large uv run main.py

True
chat-large
true
sk-test...

load_dotenv does not replace variables that are already set. That is deliberate and important: .env is only a convenience for your own machine, and the real environment set on a server or in a container always wins. (load_dotenv(path, override=True) reverses this order, but there is almost never a reason to want that.)

Notice also that load_dotenv does not raise when the file is missing — it just returns False. So never assume ".env was read"; check the values you need separately, as the next example does.

Two rules that every professional repository follows:

• .env is never committed. It goes in .gitignore. Once a key has entered Git history it stays there even after the file is deleted; the only safe response is to revoke the key and issue a new one.
• .env.example is committed. It has the same variable names with empty or harmless values. A new colleague copies it to .env and knows exactly which values are needed.

A complete example

Now all of it together: a config loader for the support bot. The project was created with uv init --app --python 3.12 support-bot, then uv add pyyaml python-dotenv. The layout:

support-bot/
├── .env
├── .env.example
├── config/
│   └── app.yaml
├── prompts/
│   ├── summarise.txt
│   └── system.txt
├── pyproject.toml
├── settings.py
└── uv.lock

config/app.yaml — committed, nothing secret in it:

app:
  name: support-bot
  debug: false
llm:
  model: chat-small
  temperature: 0.2
  max_tokens: 512
retrieval:
  top_k: 4

.env — not committed:

# Local values for this machine. Never commit this file.
LLM_API_KEY=sk-test-placeholder-1234
APP_DEBUG=true

.env.example — committed:

# Copy to .env and fill in. This file is committed; .env is not.
LLM_API_KEY=
APP_DEBUG=false

prompts/system.txt holds one line, You are a support assistant. Answer only from the provided context., and prompts/summarise.txt holds Summarise the ticket below in two sentences.

And settings.py:

import os
import sys
from pathlib import Path

import yaml
from dotenv import load_dotenv

PROJECT_ROOT = Path(__file__).resolve().parent
CONFIG_FILE = PROJECT_ROOT / "config" / "app.yaml"
PROMPTS_DIR = PROJECT_ROOT / "prompts"


class ConfigError(Exception):
    """Configuration is missing or invalid."""


def parse_bool(value: str) -> bool:
    normalized = value.strip().lower()
    if normalized in {"1", "true", "yes", "on"}:
        return True
    if normalized in {"0", "false", "no", "off"}:
        return False
    raise ConfigError(f"not a boolean: {value!r}")


def load_yaml(path: Path) -> dict:
    if not path.is_file():
        raise ConfigError(f"config file not found: {path.relative_to(PROJECT_ROOT).as_posix()}")
    data = yaml.safe_load(path.read_text(encoding="utf-8"))
    if not isinstance(data, dict):
        raise ConfigError(f"{path.name} must contain a mapping at the top level")
    return data


def load_prompts(folder: Path) -> dict[str, str]:
    return {path.stem: path.read_text(encoding="utf-8").strip() for path in sorted(folder.glob("*.txt"))}


def require_env(name: str) -> str:
    value = os.getenv(name, "").strip()
    if not value:
        raise ConfigError(f"missing required environment variable: {name}")
    return value


def load_settings() -> dict:
    load_dotenv(PROJECT_ROOT / ".env")
    config = load_yaml(CONFIG_FILE)

    temperature = float(os.getenv("LLM_TEMPERATURE", config["llm"]["temperature"]))
    if not 0 <= temperature <= 2:
        raise ConfigError(f"LLM_TEMPERATURE must be between 0 and 2, got {temperature}")

    debug = config["app"]["debug"]
    if "APP_DEBUG" in os.environ:
        debug = parse_bool(os.environ["APP_DEBUG"])

    return {
        "app_name": config["app"]["name"],
        "debug": debug,
        "model": os.getenv("LLM_MODEL", config["llm"]["model"]),
        "temperature": temperature,
        "top_k": int(config["retrieval"]["top_k"]),
        "api_key": require_env("LLM_API_KEY"),
        "prompts": load_prompts(PROMPTS_DIR),
    }


def mask(secret: str) -> str:
    return secret[:3] + "*" * 8


if __name__ == "__main__":
    try:
        settings = load_settings()
    except ConfigError as exc:
        print(f"config error: {exc}")
        sys.exit(1)

    for key in ["app_name", "debug", "model", "temperature", "top_k"]:
        print(f"{key:<12} {settings[key]}")
    print(f"{'api_key':<12} {mask(settings['api_key'])}")
    print(f"{'prompts':<12} {', '.join(settings['prompts'])}")

Four runs — normal, with two values changed from the shell, with .env moved aside, and with an impossible temperature:

uv run settings.py
LLM_TEMPERATURE=0 LLM_MODEL=chat-large uv run settings.py
uv run settings.py
LLM_TEMPERATURE=5 uv run settings.py

app_name     support-bot
debug        True
model        chat-small
temperature  0.2
top_k        4
api_key      sk-********
prompts      summarise, system
app_name     support-bot
debug        True
model        chat-large
temperature  0.0
top_k        4
api_key      sk-********
prompts      summarise, system
config error: missing required environment variable: LLM_API_KEY
config error: LLM_TEMPERATURE must be between 0 and 2, got 5.0

(Before the third run the .env file was renamed out of the way, and it was put back before the fourth.) The last two runs end with exit code 1 and the first two with 0, so a deploy script or CI can easily tell that the config is wrong.

The decisions behind this loader:

Every path comes from PROJECT_ROOT. Path(__file__).resolve().parent is computed once and every other path is built on it. Whatever folder the program is started from, it reads the same files.

One explicit order of precedence. YAML defaults first, .env above them, and the real environment of the shell or server above everything. In the second run the model and temperature changed without editing a single file — which is how config is changed in one place in production.

temperature 0.0 survives. os.getenv("LLM_TEMPERATURE", ...) only falls back to the YAML value when the variable is truly absent — there is no or anywhere. Likewise debug checks "APP_DEBUG" in os.environ, not whether the value is truthy.

Type conversion and range checks in one place. float(), int(), parse_bool() and the temperature's 0 <= temperature <= 2 all live in load_settings. The rest of the program can assume settings["temperature"] is a valid number.

A dedicated ConfigError. Every config problem surfaces as one kind of error, and the __main__ block turns it into a clear one-line message — which value to fix, instead of a traceback.

Secrets are never printed in full. mask shows the first three characters, so you can tell which kind of key was loaded, but the key never reaches the logs. The mask length is fixed too, so nothing can be learned from the key's length.

Prompt files via glob. Drop a new .txt into prompts/ and it appears in settings["prompts"] without touching the code. sorted() is there because the order glob returns files in depends on the operating system.

One limitation: load_settings returns a plain dictionary, so a typo like settings["tempreature"] is only caught as a KeyError at run time, and every new setting needs its conversion and check written by hand. A tool that does this declaratively is coming later in the course; the principles here — load in one place, the right types, stop at the start — stay the same with it.

When it breaks

FileNotFoundError: [Errno 2] No such file or directory: 'config\\app.json' The path is relative and the program was started from another folder. Build the path from Path(__file__).resolve().parent. To see which folder the lookup starts from, print Path.cwd().

TypeError: unsupported operand type(s) for /: 'str' and 'str' "config" / "app.json" — strings on both sides. Make the leftmost part a Path: Path("config") / "app.json".

json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 36 (char 35) The JSON has a trailing comma after its last item ({"top_k": 4,}), or a key in single quotes. Python dictionaries accept these; JSON does not. The column number in the message shows exactly where the parser got stuck.

TypeError: load() missing 1 required positional argument: 'Loader' Old code calling yaml.load(text). Write yaml.safe_load(text) instead.

KeyError: 'LLM_API_KEY' The code read os.environ["LLM_API_KEY"] and the variable is not set. Check that .env is in the project root, that load_dotenv got the right path, and that the name is spelt the same everywhere. To give a clear message for missing values, use a function like require_env above.

ModuleNotFoundError: No module named 'dotenv' python-dotenv was never added to the project, or the program was run without uv run, so the project's environment was not used. Run uv add python-dotenv, and start the program with uv run.

Your turn

Write a config loader that layers an environment-specific config on top of a base config. Create a project with uv init --app --python 3.12 config-lab, run uv add python-dotenv, then create these files:

• config/base.json: {"llm": {"model": "chat-small", "temperature": 0.2}, "retrieval": {"top_k": 4}}
• config/staging.json: {"llm": {"temperature": 0}, "retrieval": {"top_k": 8}}
• .env: one line, APP_ENV=staging

Write main.py so that:

1. Every path is built from Path(__file__).resolve().parent, so the program gives the same result from any folder
2. APP_ENV comes from the environment or .env, defaulting to dev; for an environment with no file (such as config/prod.json) the program prints config error: no config/prod.json and stops with exit code 1
3. The two configs are merged deeply: nested dictionaries merge key by key instead of being replaced whole, so base.json's model survives even though staging.json has none
4. A temperature of 0 stays 0 — no or

Running uv run main.py from the project root should give exactly this:

env          staging
model        chat-small
temperature  0
top_k        8

Then test two things. First, run APP_ENV=prod uv run main.py and check that the program stops with a clear message. Second, try a plain {**base, **overlay} instead of the deep merge and print config["llm"] — where did model go, and why did the merge line itself raise no error?