FreeCampus Python

Keep Text Intact Across Encodings and Newlines

Distinguish characters from bytes, diagnose decoding failures, normalize comparisons carefully, and preserve multilingual text and line boundaries.
python-foundations files-paths-external-data unicode encodings
Open in Colab
  • Level: Python Foundations
  • Estimated time: 4–5 hours
  • You will learn: Explain the character–encoding–byte boundary, encode and decode explicitly, diagnose failures without hiding damage, compare normalized Unicode safely, control newline behavior, and replace a text artifact only after verification.
  • Practice in: Google Colab, JupyterLab, or a local editor using temporary multilingual fixtures

Open this lesson in Google Colab

A subtitle team exchanges lines such as "Café déjà vu", "Olá, Lua", and "東京へようこそ". One machine displays them correctly. Another shows replacement symbols. A third reads every line but produces extra blank lines after export. The words did not change; assumptions at the text boundary did.

Use these questions throughout the lesson:

1. Text and bytes are different kinds of value

Python’s str stores Unicode text. It is a sequence of characters as Python represents them, not a sequence of UTF-8 bytes. bytes stores integers from 0 to 255. An encoding defines how text maps to bytes; decoding applies that contract in reverse.

message = "Café ☕"
encoded = message.encode("utf-8")
decoded = encoded.decode("utf-8")

print(message)
print(encoded)
print("characters:", len(message))
print("bytes:", len(encoded))
print("round trip:", decoded == message)

The accented letter and coffee symbol need more than one UTF-8 byte, so byte length can exceed character length. The byte representation may display printable ASCII directly and other values as hexadecimal escapes. That display is evidence about bytes, not corrupted text.

Encoding and decoding cross an explicit boundary. The same codec must be part of both sides’ contract for a reliable round trip.

flowchart LR
  text["Python str: characters"] -->|"encode with UTF-8"| data["bytes: integer values"]
  data --> storage["file or network"]
  storage --> incoming["bytes"]
  incoming -->|"decode with UTF-8"| restored["Python str"]

A file does not carry a universal promise that it is UTF-8. The producer, protocol, format, or explicit metadata must establish the encoding.

Python prevents accidental mixing:

text = "Lua"
data = b"Lua"

try:
    print(text + data)
except TypeError as error:
    print(type(error).__name__, error)

Decide whether the operation is a text operation or a byte operation, then convert exactly at the documented boundary.

A code point is the numbered entry assigned to a Unicode character. ord shows the number and unicodedata.name supplies a descriptive name when one is defined:

import unicodedata

for character in ["A", "é", "月", "🚀"]:
    print(
        repr(character),
        f"U+{ord(character):04X}",
        unicodedata.name(character),
        character.encode("utf-8").hex(" "),
    )

This comparison makes three measurements distinct: one displayed character, one numbered code point, and one or more UTF-8 bytes. Human-perceived symbols can be more complicated still: a flag, family emoji, or accented grapheme may contain several code points. Do not promise that len(text) counts what a user would call visible characters.

2. An explicit encoding makes the file contract portable

The default text encoding can vary across platforms, locales, and Python configuration. State the encoding when reading or writing durable data:

from pathlib import Path
from tempfile import TemporaryDirectory

with TemporaryDirectory() as temporary_name:
    workspace = Path(temporary_name)
    subtitle_path = workspace / "opening.txt"
    subtitle_path.write_text("Olá, Lua!\n東京へようこそ\n", encoding="utf-8")

    restored = subtitle_path.read_text(encoding="utf-8")
    print(restored)
    assert restored == "Olá, Lua!\n東京へようこそ\n"

Here, both operations promise UTF-8. Writing first makes the notebook example self-contained; reading proves the round trip rather than depending on an unknown file.

What happens if the declared decoding contract is wrong?

text = "Café"
utf8_data = text.encode("utf-8")

try:
    utf8_data.decode("ascii")
except UnicodeDecodeError as error:
    print("codec:", error.encoding)
    print("start:", error.start)
    print("end:", error.end)
    print("reason:", error.reason)
    print("offending bytes:", error.object[error.start:error.end])

UnicodeDecodeError identifies the attempted codec and byte position. Preserve that evidence. Repeatedly trying random encodings until one produces characters can yield plausible-looking but false text.

The absence of an exception does not prove the encoding guess was correct. These Latin-1 bytes decode under a different single-byte codec into the wrong characters, then round-trip perfectly under that wrong assumption:

utf8_data = "Café".encode("utf-8")
wrong_text = utf8_data.decode("latin-1")
wrong_round_trip = wrong_text.encode("latin-1")

print(wrong_text)
assert wrong_round_trip == utf8_data

The display becomes the familiar mojibake Café, yet the bytes round-trip. A round-trip verifies internal consistency with a chosen codec; it cannot discover the producer’s contract. Use metadata, a protocol, or knowledge of the source. Re-encoding mojibake repeatedly can make recovery harder, so retain the original bytes while investigating.

Encoding can fail too:

try:
    "東京".encode("ascii")
except UnicodeEncodeError as error:
    print(type(error).__name__, error)

ASCII has no representation for those characters. The solution is normally to choose the correct output encoding—often UTF-8—not delete the characters.

3. Error policies trade fidelity for continuation

The default errors="strict" raises when conversion cannot satisfy the codec. That is the safest default because it stops at the boundary instead of hiding loss.

invalid_utf8 = b"signal:\xff:end"

for policy in ["replace", "backslashreplace", "ignore"]:
    result = invalid_utf8.decode("utf-8", errors=policy)
    print(policy, repr(result))

Typical results illustrate three policies:

  • replace inserts �, visibly marking damage;
  • backslashreplace exposes the offending byte as text such as \\xff;
  • ignore drops information and can join text that was never adjacent.
WarningContinuation is not correctness

Use a non-strict policy only when the application contract says how data loss is recorded and reviewed. errors="ignore" should not be a reflexive “fix” for an unknown input. Keep the original bytes so another tool or corrected contract can recover them later.

Checkpoint: protect the character–byte contract

4. Byte-order marks require a deliberate policy

Some UTF-8 text begins with the byte sequence EF BB BF, commonly called a UTF-8 byte-order mark (BOM). It is unnecessary for UTF-8 byte order, but tools sometimes emit it as a signature. Decoding with plain utf-8 exposes a leading Unicode character \ufeff; utf-8-sig consumes it when present.

with_bom = b"\xef\xbb\xbfheadline\n"

print(repr(with_bom.decode("utf-8")))
print(repr(with_bom.decode("utf-8-sig")))

This matters when the first CSV header unexpectedly becomes "\ufeffname" instead of "name". If your input contract permits a UTF-8 BOM, opening with encoding="utf-8-sig" handles files with or without it. Do not strip \ufeff from arbitrary positions; the policy belongs at the start of a documented text stream.

5. Visually identical text can use different code points

Unicode sometimes represents the same visible text in more than one way. é can be one precomposed code point or an e followed by a combining accent.

import unicodedata

composed = "Café"
decomposed = "Cafe\u0301"

print(composed == decomposed)
print([f"U+{ord(character):04X}" for character in composed])
print([f"U+{ord(character):04X}" for character in decomposed])

normalized_left = unicodedata.normalize("NFC", composed)
normalized_right = unicodedata.normalize("NFC", decomposed)
print(normalized_left == normalized_right)

Normalization form NFC is useful when an application wants canonically equivalent text to compare equal. NFD decomposes compatible sequences instead. Other forms, NFKC and NFKD, perform compatibility transformations that can change distinctions; do not choose them without a domain reason.

Preserve original display text when evidence matters:

record = {
    "display_name": decomposed,
    "comparison_name": unicodedata.normalize("NFC", decomposed),
}

assert record["display_name"] == decomposed
assert record["comparison_name"] == unicodedata.normalize("NFC", composed)

Normalization is not case folding, translation, spell checking, or security validation. It answers one narrower representation question.

6. Newlines have stored and in-memory forms

Common line terminators include:

  • line feed: \n, conventional on Unix-like systems;
  • carriage return plus line feed: \r\n, conventional on Windows;
  • carriage return: \r, present in some older data.

repr makes invisible separators visible:

mixed = "north\r\nsouth\nwest\r"
print(repr(mixed))
print(mixed.splitlines())

splitlines() understands several line boundaries and removes them by default. Use split("\n") only when the contract specifically requires a line-feed separator and you want its exact empty-field behavior.

When a text file is opened with newline=None (the default), Python applies universal newline translation on reading: \r, \r\n, and \n become \n in the returned text. The handle’s .newlines attribute reports terminators it has observed:

from pathlib import Path
from tempfile import TemporaryDirectory

with TemporaryDirectory() as temporary_name:
    path = Path(temporary_name) / "mixed-lines.txt"
    path.write_bytes(b"alpha\r\nbeta\ngamma\r")

    with path.open(encoding="utf-8", newline=None) as handle:
        translated = handle.read()
        observed = handle.newlines

    with path.open(encoding="utf-8", newline="") as handle:
        preserved = handle.read()

    print("translated:", repr(translated))
    print("observed:", observed)
    print("preserved:", repr(preserved))

With newline="", line endings are returned without translation. This is the setting required by Python’s CSV module because the CSV parser owns record newline handling. For ordinary application text, translation is often useful.

Be precise when removing terminators. line.rstrip() removes all trailing whitespace, including spaces and tabs that may be meaningful. rstrip("\r\n") removes any run of those characters, while splitlines() expresses the intent to divide lines. Choose according to the data contract.

Compare a subtitle whose indentation belongs to its display:

indented_subtitle = "    whisper from the left    \r\n"

print("strip:", repr(indented_subtitle.strip()))
print("newline only:", repr(indented_subtitle.rstrip("\r\n")))

The unqualified strip() removes both leading and trailing spaces as well as the terminator. That may be correct for an identifier field and destructive for preformatted text. The operation name alone does not define the policy; the field contract does.

A short matrix makes newline choices easier to review:

Operation Preserves terminator? Preserves surrounding spaces? Typical use
splitlines() no yes divide ordinary text into logical lines
splitlines(keepends=True) yes yes inspect or reproduce exact boundaries
rstrip("\r\n") no trailing CR/LF run yes remove a known record terminator
strip() no no normalize a field only when all edge whitespace is insignificant

A final newline is also a policy. Many line-oriented tools expect one. Build it deliberately:

subtitle_lines = ["Olá, Lua!", "東京へようこそ"]
subtitle_text = "\n".join(subtitle_lines) + "\n"
assert subtitle_text.endswith("\n")

Checkpoint: compare and split text deliberately

7. Replace an output only after creating and checking it

Writing directly to a known-good destination can leave it empty or partial if generation fails. A safer local workflow writes a sibling temporary artifact, reads it back under the promised contract, and only then replaces the final path.

from pathlib import Path


def replace_verified_text(path, text):
    """Write verified UTF-8 text beside path, then replace path."""
    path = Path(path)
    temporary_path = path.with_name(path.name + ".tmp")

    try:
        temporary_path.write_text(text, encoding="utf-8", newline="\n")
        restored = temporary_path.read_text(encoding="utf-8")
        if restored != text:
            raise OSError("temporary text did not round-trip")
        temporary_path.replace(path)
    except Exception:
        temporary_path.unlink(missing_ok=True)
        raise

    return path

Path.replace() requests a filesystem replacement. Keeping the temporary file in the same directory avoids an unnecessary cross-filesystem move. This pattern reduces the window in which the destination is incomplete, but it is not a universal durability or transaction guarantee. Filesystem, operating-system, power-loss, permissions, locking, and directory-sync behavior matter in high-stakes systems. State the scope of the promise you have actually tested.

Exercise both paths:

from tempfile import TemporaryDirectory

with TemporaryDirectory() as temporary_name:
    output_path = Path(temporary_name) / "subtitles.txt"
    output_path.write_text("previous good version\n", encoding="utf-8")

    new_text = "Olá, Lua!\n東京へようこそ\n"
    replace_verified_text(output_path, new_text)

    assert output_path.read_text(encoding="utf-8") == new_text
    assert not (output_path.parent / "subtitles.txt.tmp").exists()

For JSON, Lesson 3 will verify that the temporary text parses back to the expected Python structure, a stronger check than text equality alone.

8. Restore the multilingual subtitle reel

Build a lab with four supplied byte payloads:

subtitle_payloads = {
    "opening-utf8.txt": "Olá, Lua!\n".encode("utf-8"),
    "tokyo-bom.txt": b"\xef\xbb\xbf" + "東京へようこそ\r\n".encode("utf-8"),
    "cafe-decomposed.txt": "Cafe\u0301\n".encode("utf-8"),
    "damaged.txt": b"signal:\xff:end\n",
}

Your tasks:

  1. Write the payloads as bytes in a temporary incoming directory.
  2. Decode the first three with a documented UTF-8/BOM policy and preserve their original display strings.
  3. Store NFC comparison forms separately and prove the decomposed café matches "Café" after normalization.
  4. Use repr to record the translated and preserved line endings of the BOM file.
  5. Attempt strict decoding of damaged.txt; capture codec, position, and offending bytes in a rejection record without deleting the original.
  6. Join accepted display lines with \n, add one final newline, and call replace_verified_text.
  7. Rerun from a clean workspace and compare exact output text.
Inspect one recovery loop after completing your investigation 
import unicodedata
from pathlib import Path
from tempfile import TemporaryDirectory

with TemporaryDirectory() as temporary_name:
    root = Path(temporary_name)
    incoming = root / "incoming"
    incoming.mkdir()
    for name, payload in subtitle_payloads.items():
        (incoming / name).write_bytes(payload)

    accepted = []
    rejected = []
    for path in sorted(incoming.glob("*.txt")):
        try:
            display_text = path.read_text(encoding="utf-8-sig")
        except UnicodeDecodeError as error:
            rejected.append(
                {
                    "source": path.name,
                    "encoding": error.encoding,
                    "position": error.start,
                    "bytes": error.object[error.start:error.end].hex(),
                }
            )
            continue

        accepted.append(
            {
                "source": path.name,
                "display": display_text,
                "comparison": unicodedata.normalize("NFC", display_text),
            }
        )

    cafe = next(item for item in accepted if item["source"] == "cafe-decomposed.txt")
    assert cafe["comparison"].strip() == "Café"
    assert rejected[0]["source"] == "damaged.txt"

    output_text = "".join(item["display"].replace("\r\n", "\n") for item in accepted)
    if output_text and not output_text.endswith("\n"):
        output_text += "\n"
    output_path = replace_verified_text(root / "restored.txt", output_text)
    assert output_path.read_text(encoding="utf-8") == output_text
    saved_bytes = output_path.read_bytes()
    assert saved_bytes.decode("utf-8") == output_text
    assert saved_bytes.endswith(b"\n")

The original bytes remain available. Accepted records preserve display text and store a separate normalized comparison value. The rejection explains why one payload could not cross the declared boundary. Inspecting the saved bytes closes the whole contract: Python text was encoded as UTF-8, the result decodes back to the intended text, and the byte artifact retains the chosen final newline.

Checkpoint: publish text without hiding damage

9. Key points for text integrity

  • str contains Unicode text; bytes contains integer byte values. Encoding and decoding cross between them under an explicit codec contract.
  • A decoding exception provides useful byte-position evidence. Do not replace it with silent loss unless the application explicitly records that trade-off.
  • utf-8-sig is useful when a producer may place a UTF-8 BOM at the beginning of a text stream.
  • Unicode normalization can align canonically equivalent comparison forms while original display text remains preserved.
  • Newline translation, splitting, stripping, and final-newline choices express different policies; choose them deliberately.
  • Create and verify a sibling temporary artifact before replacement when a previous good local output should survive generation failures.

10. References and next step

Next, apply these encoding and newline contracts to structured records. CSV, JSON, and YAML each preserve a different set of information and require their own parser and writer rather than ad hoc string operations.

Back to top