I’m reading Robots don’t take people’s jobs – they make new ones via the New Scientist https://t.co/jkptO2VRuZ
From my twitter post: Uber self driving car crash: I’m not convinced a human driver would have avoided this given the dash cam footage https://t.co/rm0gb4LzlQ
Uber self driving car crash: I’m not convinced a human driver would have avoided this given the dash cam footage https://t.co/rm0gb4LzlQ
From my twitter post: US spy lab hopes to geotag every outdoor photo on social media https://t.co/UkMVkMAKWp
US spy lab hopes to geotag every outdoor photo on social media https://t.co/UkMVkMAKWp
Piping numpy arrays to other processes in python
To pipe data from one process to another as a stream in python we need to pickle the object and pass it to the pipe stream.
In this example I’ve used Numpy arrays but this could be applied to any object that can be pickled in Python.
This took far too long to get working and I could find little information online on how to put it all together so here it is.
This code is Python 3 only, I’ve only run this on a Mac.
I’ve used binary as the stream rather than text purley becuase of effiencies. Numpy arrays can get huge! This means readline()
is not
going to work. Instead, I send a single control byte , 1, for data and 0 for stop. This could be extended to include other control operations.
I then send the length of the data as a 8 byte int, followed by the data itself.
import numpy as np
import pickle
import sys
import io
import time
#define some control bytes
control_data=bytes([1])
control_stop=bytes([0])
def send_data(arr):
dataStr=pickle.dumps(arr) #pickle the data array into a byte array
dlen=len(dataStr).to_bytes(8, byteorder='big') #find the length of the array and
print(control_data.decode('latin-1'),end='',flush=True) #convert this to a byte array
print(dlen.decode('latin-1'), end='', flush=True) #encode the data and write it
print(dataStr.decode('latin-1'), end='', flush=True) # end='' will remove that extra \r\n
def send_stop():
print(control_stop.decode('latin-1'), end='', flush=True)
#set the stdout such that it prints in latin-1, sys.stdout.detach() is a binary stream
sys.stdout = io.TextIOWrapper(sys.stdout.detach(), encoding='latin-1')
for p in range(10):
arr=np.ones((5000,500))*p #generate some data
send_data(arr)
#the sleep is purely for testing and can be removed, ie does the reader fall over after a long delay
time.sleep(.1)
send_stop()
import numpy as np
import sys
import pickle
#define some control bytes
control_data=bytes([1])
control_stop=bytes([0])
while True:
data=sys.stdin.buffer.read(1) #read the control byte
if data==control_data:
data=sys.stdin.buffer.read(8) #read the data length
dlen=int.from_bytes(data, byteorder='big')
print('data lenght %d'%dlen)
data=sys.stdin.buffer.read(dlen) #read the data
npd=pickle.loads(data) #unpickle
print(npd.shape)
print(npd.max())
elif data==control_stop:
print('stopped')
break
else:
print('Oh no')
to run this
python simpleSend.py | python simpleReceiver.py
If we want to use Python’s subprocess module to start simpleReceiver.py
we basically need to write
to the STDIN instead of print
import numpy as np
import pickle
import sys
import subprocess as sp
#define some control bytes
control_data=bytes([1])
control_stop=bytes([0])
def send_data(arr,buff):
dataStr=pickle.dumps(arr) #pickle the data array into a byte array
dlen=len(dataStr).to_bytes(8, byteorder='big') #find the length of the array and
mp.stdin.write(control_data)
mp.stdin.write(dlen)
mp.stdin.write(dataStr)
mp.stdin.flush() #not sure this needed
def send_stop(mp):
mp.stdin.write(control_stop)
mp.stdin.flush()
try:
mp = sp.Popen("python3 simpleReceiver.py", shell = True,stdin=sp.PIPE)
except sp.CalledProcessError as err:
print('ERROR:', err)
sys.exit(-1)
for p in range(10):
arr=np.ones((5000,5000))*p #generate some data
send_data(arr,mp)
send_stop(mp)
With such a large array 5000×5000 this takes sometime. Running it through the python profiler indicates about 75% of the time is taken by pickle.dumps
and most of the rest of the remaining 25% is taken by the write operation. Numpy’s own method gives a speed increase. Replacing dataStr=pickle.dumps(arr)
with dataStr=arr.tobytes()
and npd=pickle.loads(data)
with npd=np.frombuffer(data)
more than halves the time taken but lose the shape and dtype information. This would have to be sent along with the data.
From my twitter post: Add your name to the open letter to Universities engaging in punitive action against staff during the UCU strike an… https://t.co/u5M38lfO1s
Add your name to the open letter to Universities engaging in punitive action against staff during the UCU strike an… https://t.co/u5M38lfO1s
From my twitter post: China: Big Data Fuels Crackdown in Minority Region https://t.co/9cU9EDFcKV
China: Big Data Fuels Crackdown in Minority Region https://t.co/9cU9EDFcKV
From my twitter post: I’ve signed the petition calling on Jeremy Hunt to stop the use of financial incentives to cut GP referrals – will… https://t.co/Acxqk51ndj
I’ve signed the petition calling on Jeremy Hunt to stop the use of financial incentives to cut GP referrals – will… https://t.co/Acxqk51ndj
From my twitter post: One on my students!! Well done. Iraqi government recognises impact of Sussex researcher https://t.co/AzjVzAwWRW via @SussexUni
One on my students!! Well done. Iraqi government recognises impact of Sussex researcher https://t.co/AzjVzAwWRW via @SussexUni
From my twitter post: Face-recognition software is perfect – if you’re a white man https://t.co/k14Ui5up3S
Face-recognition software is perfect – if you’re a white man https://t.co/k14Ui5up3S
Finding webcams
List all devices on windows using ffmpeg
ffmpeg -list_devices true -f dshow -i dummy
and on Linux
v4l2-ctl –list-devices
To get the device capabilites
ffmpeg -f dshow -list_options true -i video="Mobius"
where “Mobius” is the name of the camera.
On the Mac use
ffmpeg -f avfoundation -list_devices true -i ""