python - Can you specify a buffer size when opening a file in text IO mode - Stack Overflow

I'm confused by the documentation regarding the buffering when reading a file opened with the open function.

Does the buffering parameter do anything when mode is r so that the file is opened in text I/O mode?

The docs state:

Note that specifying a buffer size this way applies for binary buffered I/O, but TextIOWrapper (i.e., files opened with mode='r+') would have another buffering.

What does another mean? An additional buffering or a different buffering?

I'd assume it's the latter, which means that the buffering parameter does not specify the size in bytes of a fixed-size chunk buffer. Why would they state that the parameter has a different meaning but not explain it?

What's the conclusion to this?

open('myfile.txt', 'r', 1024)

Does this use a buffer of 1024 bytes? Does the parameter change anything in this mode?

I'm confused by the documentation regarding the buffering when reading a file opened with the open function.

Does the buffering parameter do anything when mode is r so that the file is opened in text I/O mode?

The docs state:

Note that specifying a buffer size this way applies for binary buffered I/O, but TextIOWrapper (i.e., files opened with mode='r+') would have another buffering.

What does another mean? An additional buffering or a different buffering?

I'd assume it's the latter, which means that the buffering parameter does not specify the size in bytes of a fixed-size chunk buffer. Why would they state that the parameter has a different meaning but not explain it?

What's the conclusion to this?

open('myfile.txt', 'r', 1024)

Does this use a buffer of 1024 bytes? Does the parameter change anything in this mode?

• python
• python-3.x

Share Improve this question Follow edited Jan 31 at 10:49 T3rm1 asked Jan 31 at 9:17 T3rm1T3rm1 2,60455 gold badges4040 silver badges5353 bronze badges 7

• The details of how buffering is used is fully explained in the following paragraphs in the link you provided above. – OldBoy Commented Jan 31 at 9:38
• 1 I think I understand that TextIOWrapper has additional and different buffering in addition to the buffering offered open(). I.e. TextIOWrapper also buffers. I would guess open buffers bytes, TextIOWrapper buffers characters. – KamilCuk Commented Jan 31 at 9:39
• In addition to the doc that you have linked, you should read the docs on io. – kapad Commented Jan 31 at 10:23
• @T3rm1 seems like the docs are outdated. You should open an issue or a PR to get them fixed. – kapad Commented Jan 31 at 12:02
• source code for the docs is here and the source for the built-in function is here after following the wrapper defined here. Check the comments for the open method. The caveat regarding buffer size that is present in the docs, is not present in the function comments. – kapad Commented Jan 31 at 12:07

 |  Show 2 more comments

1 Answer 1

Sorted by: Reset to default 0

TL;DR

Yes. The size of the buffer that is wrapped by TextIOWrapper would be 1024 bytes (or characters or lines? - I'm not a 100% certain).

Long version

I think the python docs are outdated in this case.

pylifecycle.c (init_set_builtins_open)

source

This method redirects the builtin open to io.open. So, the next place to dig in is io.open.

Lib/_pyio.py (open)

source

This method is calling os builtins to open the file via instantiation of class FileIO(RawIOBase).

If you continue to read the open method, it's passing in the buffer size to the wrapped Buffered(Reader/Writer/Random).

So, the 1024 buffer size that's being passed would be propagated to the wrapped Buffered object. This buffer size may represent the size in bytes (or characters or lines? - I'm not a 100% certain).