Buffering and Flushing in Python

When working with files in Python, data is often not written directly to disk immediately. Instead, it is temporarily stored in a buffer. This improves performance, especially for large I/O operations. Understanding buffering and how to flush data manually is crucial for reliability in file handling.

Key Concepts

• Buffering: Temporary storage area for data before it’s written to disk.
• flush(): Forces the buffer to write its contents to the file immediately.
• When buffers flush:
  • When the buffer is full
  • When the file is closed
  • When flush() is called
• Buffered vs Unbuffered: Use buffering=0 for unbuffered, 1 for line-buffered, or specify an integer buffer size.

10 Python Examples for Buffering and Flushing

Example 1: Writing with default buffering

with open('log.txt', 'w') as f:
    f.write('Log started\\n')
    # Automatically buffered, flushes on close

Example 2: Forcing flush manually

with open('log.txt', 'w') as f:
    f.write('Important data\\n')
    f.flush()  # forces immediate write

Example 3: Writing with no buffering (buffering=0)

# Binary mode required for buffering=0
with open('data.bin', 'wb', buffering=0) as f:
    f.write(b'hello')

Example 4: Line buffered writing (buffering=1)

with open('data.txt', 'w', buffering=1) as f:
    f.write("line1\\n")  # Flushes on newline

Example 5: Writing with a custom buffer size

with open('buffered.txt', 'w', buffering=8192) as f:
    f.write('Buffered write with 8KB buffer')

Example 6: When flush() is needed in long-running programs

import time

with open('progress.log', 'w') as f:
    for i in range(5):
        f.write(f"Step {i}\\n")
        f.flush()  # ensures each step is written immediately
        time.sleep(1)

Example 7: Using flush() with sys.stdout

import sys
import time

for i in range(3):
    print(f"Progress {i}", end=' ', flush=True)
    time.sleep(1)

Example 8: Writing and checking data before file close

with open('temp.txt', 'w') as f:
    f.write('Not flushed yet')
    f.flush()  # ensures data is on disk now

Example 9: Using try-finally to ensure flush

f = open('output.txt', 'w')
try:
    f.write('Critical log entry\\n')
    f.flush()
finally:
    f.close()

Example 10: Comparing buffered vs unbuffered timing

import time

start = time.time()
with open('big.txt', 'w', buffering=1024) as f:
    for _ in range(10000):
        f.write('x' * 100 + '\\n')
print("Buffered:", time.time() - start, "sec")