Python: Os.read() Blocking On Fd

How to avoid the problem of line buffering if the fd is a bash shell?

Solution 1:

Here's a summary of what I've understood from your question and comments plus an unsatisfactory workaround at the end of the answer. The summary should help others to answer your question.

It seems you have a program (non-python) that wants to execute arbitrary shell commands that might require a tty (Q: Why not just use a pipe (popen())?) and you have chosen as a solution to execute a Python script that runs pty.spawn("/bin/sh", read_callback) and write/read to its stdin/stdout.

And now you have a problem that you can't find the end of output for a shell command i.e., you don't know when to stop reading program.py stdout and if you try to read too much then your non-python program blocks.

First of all, os.read inside read_callbackdoes not block. It may return less than 1024 bytes but it doesn't block:

def read_callback(fd):
    data = os.read(fd, 1024) # <-- this doesn't block
    return data 

Though it doesn't help when the parent program tries to read.

Workaround

To avoid blocking on read, the parent can change PS1 environment in the running sh to some unique value or inject echo something_unique after each command. Then it can read one byte at a time until it reads something_unique. It should stop at this point.

As an alternative you could try to make the pipes nonblocking in your parent and read more than one byte at a time. Or use an exchange protocol that defines clear message boundaries and run shell commands one by one e.g., using pexpect.run() (to detect the end of output easily in the Python side).

Solution 2:

This seems to work:

#!/usr/local/cpython-3.3/bin/pythonimport os
import pty
#mport sysimport fcntl

OFLAGS = Nonedefset_nonblocking(file_handle):
    """Make a file_handle non-blocking."""global OFLAGS
    OFLAGS = fcntl.fcntl(file_handle, fcntl.F_GETFL)
    nflags = OFLAGS | os.O_NONBLOCK
    fcntl.fcntl(file_handle, fcntl.F_SETFL, nflags)


defmain():
    (pid, file_handle) = pty.fork()
    if pid == 0:
        # we're in the child
        os.execl('/bin/sh', '/bin/sh', '-i')
    else:
        #file_handle = os.open('/dev/null', os.O_RDONLY)
        set_nonblocking(file_handle)
        whileTrue:
            try:
                # return 1-n bytes or exception if no bytes
                data = os.read(file_handle, 1024)
            except BlockingIOError:
                #sys.stdout.write('no data read\r')passelse:
                print(len(data), data)

main()

Sometimes the way to deal with nonblocking I/O, is to use a thread or subprocess, BTW. Then one thread or subprocess can block while others merrily do their work.

HTH