How can I read large text files line by line, without loading them into memory?

--------------------------------------------------
Rise to the top 3% as a developer or hire one of them at Toptal: https://topt.al/25cXVn
--------------------------------------------------

Music by Eric Matyas
https://www.soundimage.org
Track title: Techno Bleepage Open

--

Chapters
00:00 How Can I Read Large Text Files Line By Line, Without Loading Them Into Memory?
00:21 Accepted Answer Score 447
00:41 Answer 2 Score 81
01:04 Answer 3 Score 22
01:25 Answer 4 Score 21
01:38 Answer 5 Score 19
01:49 Thank you

--

Full question
https://stackoverflow.com/questions/6475...

--

Content licensed under CC BY-SA
https://meta.stackexchange.com/help/lice...

--

Tags
#python #fileio

#avk47

ACCEPTED ANSWER

Score 448

Use a for loop on a file object to read it line-by-line. Use with open(...) to let a context manager ensure that the file is closed after reading:

with open("log.txt") as infile:
    for line in infile:
        print(line)

ANSWER 2

Score 81

All you need to do is use the file object as an iterator.

for line in open("log.txt"):
    do_something_with(line)

Even better is using context manager in recent Python versions.

with open("log.txt") as fileobject:
    for line in fileobject:
        do_something_with(line)

This will automatically close the file as well.

ANSWER 3

Score 23

You are better off using an iterator instead.
Relevant: fileinput — Iterate over lines from multiple input streams.

From the docs:

import fileinput
for line in fileinput.input("filename", encoding="utf-8"):
    process(line)

This will avoid copying the whole file into memory at once.

ANSWER 4

Score 20

An old school approach:

fh = open(file_name, 'rt')
line = fh.readline()
while line:
    # do stuff with line
    line = fh.readline()
fh.close()