Implementing PKCS#7 Padding on a Stream of Unknown Length?

Question

I have a fairly simple Python program using PyCrypto to use AES+CBC to encrypt a stream of input. In order to adhere to the 16-byte input size multiple, I've implemented PKCS#7 by hand. (While I know implementing cryptographic methods by hand is generally a bad idea, padding 1-16 bytes of data isn't exactly the hardest thing in the world. Here is essentially what I'm doing:

while True:
    input_chunk = infile.read(1024)

    if len(input_chunk) == 0:
        # if we've reached the end of the file and it _is_ a 
        # multiple of 16 in length, pad 16 bytes with the value '16'
        end_of_line = True
        input_chunk += struct.pack('{}B'.format(16), *(16 * [16]))
    elif len(input_chunk) % 16 > 0:
        # if we don't have an input_chunk which is divisible by 16,
        # pad it by the remainder with bytes with the value of the
        # remainder
        end_of_line = True
        input_chunk_remainder =  16 - (len(input_chunk) % 16)
        input_chunk += struct.pack('{}B'.format(input_chunk_remainder),
                *(input_chunk_remainder * [input_chunk_remainder]))

    outfile.write(encryption_cipher.encrypt(input_chunk))

    if end_of_line:
        break

The only problem I'm now having is trying to understand how I'll decrypt an input stream if I don't know its length in advance. The encryption I'll be implementing will actually be used to read a stream of bytes from a socket, decrypting each 'chunk' as it comes across.

How will I know when to trim the PKCS#7 padding? Can I assume that if the last n bytes in the stream are all the same value from 1-16 that I can trim the input after decrypting it? I would assume that this is dangerous and may trim data unnecessarily.

Is there a way to solve this problem?