How To Read Only A Chunk Of Csv File Fast?

I'm using this answer on how to read only a chunk of CSV file with pandas. The suggestion to use pd.read_csv('./input/test.csv' , iterator=True, chunksize=1000) works excellent bu

Solution 1:

pd.read_csv('./input/test.csv', iterator=True, chunksize=1000) returns an iterator. You can use the next function to grab the next one

reader = pd.read_csv('./input/test.csv', iterator=True, chunksize=1000)

next(reader)

This is often used in a for loop for processing one chunk at a time.

for df in pd.read_csv('./input/test.csv', iterator=True, chunksize=1000):
    pass