How can I read large text files line by line, without loading them into memory?

Become part of the top 3% of the developers by applying to Toptal https://topt.al/25cXVn

--

Music by Eric Matyas
https://www.soundimage.org
Track title: Cool Puzzler LoFi

--

Chapters
00:00 Question
00:28 Accepted answer (Score 424)
00:50 Answer 2 (Score 78)
01:17 Answer 3 (Score 20)
01:31 Answer 4 (Score 19)
01:45 Thank you

--

Full question
https://stackoverflow.com/questions/6475...

Accepted answer links:
[context manager]: https://docs.python.org/3/reference/data...

--

Content licensed under CC BY-SA
https://meta.stackexchange.com/help/lice...

--

Tags
#python #fileio

#avk47

ACCEPTED ANSWER

Score 448

Use a for loop on a file object to read it line-by-line. Use with open(...) to let a context manager ensure that the file is closed after reading:

with open("log.txt") as infile:
    for line in infile:
        print(line)

ANSWER 2

Score 81

All you need to do is use the file object as an iterator.

for line in open("log.txt"):
    do_something_with(line)

Even better is using context manager in recent Python versions.

with open("log.txt") as fileobject:
    for line in fileobject:
        do_something_with(line)

This will automatically close the file as well.

ANSWER 3

Score 23

You are better off using an iterator instead.
Relevant: fileinput — Iterate over lines from multiple input streams.

From the docs:

import fileinput
for line in fileinput.input("filename", encoding="utf-8"):
    process(line)

This will avoid copying the whole file into memory at once.

ANSWER 4

Score 20

An old school approach:

fh = open(file_name, 'rt')
line = fh.readline()
while line:
    # do stuff with line
    line = fh.readline()
fh.close()