Piping numpy arrays to other processes in python

To pipe data from one process to another as a stream in python we need to pickle the object and pass it to the pipe stream.
In this example I’ve used Numpy arrays but this could be applied to any object that can be pickled in Python.
This took far too long to get working and I could find little information online on how to put it all together so here it is.
This code is Python 3 only, I’ve only run this on a Mac.

I’ve used binary as the stream rather than text purley becuase of effiencies. Numpy arrays can get huge! This means readline() is not
going to work. Instead, I send a single control byte , 1, for data and 0 for stop. This could be extended to include other control operations.
I then send the length of the data as a 8 byte int, followed by the data itself.

simpleSend.py

import numpy as np
import pickle
import sys
import io
import time

#define some control bytes
control_data=bytes([1])
control_stop=bytes([0])

def send_data(arr):
    dataStr=pickle.dumps(arr)  #pickle the data array into a byte array
    dlen=len(dataStr).to_bytes(8, byteorder='big') #find the length of the array and
    print(control_data.decode('latin-1'),end='',flush=True)  #convert this to a byte array
    print(dlen.decode('latin-1'), end='', flush=True)   #encode the data and write it
    print(dataStr.decode('latin-1'), end='', flush=True)  # end='' will remove that extra \r\n

def send_stop():
    print(control_stop.decode('latin-1'), end='', flush=True) 

#set the stdout such that it prints in latin-1,   sys.stdout.detach() is a binary stream
sys.stdout = io.TextIOWrapper(sys.stdout.detach(), encoding='latin-1')

for p in range(10):
    arr=np.ones((5000,500))*p  #generate some data
    send_data(arr)
    #the sleep is purely for testing and can be removed, ie does the reader fall over after a long delay
    time.sleep(.1)
send_stop()        

simpleReceiver.py

import numpy as np
import sys
import pickle

#define some control bytes
control_data=bytes([1])
control_stop=bytes([0])

while True:
    data=sys.stdin.buffer.read(1)   #read the control byte
    if data==control_data:
        data=sys.stdin.buffer.read(8)  #read the data length
        dlen=int.from_bytes(data, byteorder='big')
        print('data lenght %d'%dlen)        
        data=sys.stdin.buffer.read(dlen) #read the data        
        npd=pickle.loads(data)  #unpickle
        print(npd.shape)
        print(npd.max())
    elif data==control_stop:
        print('stopped')
        break
    else:
        print('Oh no')

to run this
python simpleSend.py | python simpleReceiver.py

If we want to use Python’s subprocess module to start simpleReceiver.py we basically need to write to the STDIN instead of print

import numpy as np
import pickle
import sys
import subprocess as sp

#define some control bytes
control_data=bytes([1])
control_stop=bytes([0])

def send_data(arr,buff):
    dataStr=pickle.dumps(arr)  #pickle the data array into a byte array
    dlen=len(dataStr).to_bytes(8, byteorder='big') #find the length of the array and
    mp.stdin.write(control_data)
    mp.stdin.write(dlen)
    mp.stdin.write(dataStr)
    mp.stdin.flush() #not sure this needed
     
def send_stop(mp):
    mp.stdin.write(control_stop)
    mp.stdin.flush()
     
try:
    mp = sp.Popen("python3 simpleReceiver.py",  shell = True,stdin=sp.PIPE)   
except sp.CalledProcessError as err:
    print('ERROR:', err)
    sys.exit(-1)

for p in range(10):
    arr=np.ones((5000,5000))*p  #generate some data
    send_data(arr,mp)
send_stop(mp)        

With such a large array 5000×5000 this takes sometime. Running it through the python profiler indicates about 75% of the time is taken by pickle.dumps and most of the rest of the remaining 25% is taken by the write operation. Numpy’s own method gives a speed increase. Replacing dataStr=pickle.dumps(arr) with dataStr=arr.tobytes() and npd=pickle.loads(data) with npd=np.frombuffer(data) more than halves the time taken but lose the shape and dtype information. This would have to be sent along with the data.

view raw
numpyPipe.md
hosted with ❤ by GitHub

vim cheat-sheet

A minimal cheat-sheet to get by in vim (vi /gvim)

There’s no doubt that vim/vi or gvim is an incredibly powerful editor. it’s also very lightweight and fast, making it an ideal editor on the Raspberry Pi. It does however have a learning curve. This in a minimal cheat-sheet of commands. The ones in bold are I think the ones you need to learn to be able to use it at a basic level. Also try running vimtutor from the command line for a tutorial.

Esc key – Normal mode

Inserting text

a -append

i – insert

o – open (inserts line below current line, O open above current line)

r – replace (replaces the character under the cursor with the next one entered,

R – replace mode (overwrite text until Esc pressed)

ce – change (deletes to end of word and then switches to insert mode) c$ deletes the rest of the line

Deleting text

x -delete character

dw – delete word      (d2w delete two words)

de – delete from cursor to end of word

d$ – delete from cursor to end of line

dd – delete the line  (2dd delete two lines)

Undo

u – undo

U – undo all changes on a line

Put, Past and Cut

p – put the contents of the buffer (paste) (this is the last thing deleted or yanked)

v – visual selections mode (highlights text for eg deletions or write to file with :w FILENAME)

y – yank (copies highlighted text), yw – yanks word

Moving about

gg – go to start of the file

GG – go to end of the file

504G – goto line 504

/  – search, n find next, N find previous

% – find matching bracket (,[,{

0 – move to start of the line, ^ move to first non space character.

$ – move to end of the line

Status

^G – file and position status

! – execute external command e.g. !ls

Writing and Reading

:w – save the file

:w FILENAME – save as FILENAME

:r FILENAME – inserts content of file here, you can also insert output of commands eg :r !ls

:q! -quit discarding changes

: x – quit saving changes (if there are any as opposed to :wq which always saves)

Installing MySQL with macports

This is my notes for installing MySQL on mac. It’s mainly taken from https://trac.macports.org/wiki/howto/MySQL

sudo port install mysql56-server

which installs version 5.6, but you could check this was the latest version by search the output of

port search mysql

You need to enable the port by adding it to the system path, the easiest way is by

sudo port select mysql mysql56

Then set up the database

sudo -u _mysql mysql_install_db 
sudo chown -R _mysql:_mysql /opt/local/var/db/mysql56/ 
sudo chown -R _mysql:_mysql /opt/local/var/run/mysql56/ 
sudo chown -R _mysql:_mysql /opt/local/var/log/mysql56/ 

Then start the database

sudo port load mysql56-server

We need to set a root password

/opt/local/lib/mysql56/bin/mysqladmin -u root -p password 

You will be prompted for the old password, which currently blank so just press enter, then add your new password.

You can then add some basic security to the database by running

/opt/local/bin/mysql_secure_installation

If you need the server to connect to the network (including it seems the loopback localhost – although I’m sure there is away round this), you need to edit the configuration file in /opt/local/etc/mysql56/my.conf. The file simply calls the default config file, macports-default.cnf,  which stops only has skip-network in it. Don’t edit the default since an update may overwrite it, simply edit comment out the include line with a ! symbol.

To enable a C++ interface to MySQL install

sudo port install mysql5-connector-cpp

note that this also a port mysql-connector-cpp without that 5, which is broken! A half hour of my life I’ll never get back. Unfortunately this port is out of date and links against MySQL 5.1. It can still be used with 5.6 but it does mean macports will also install 5.1 on your system.

To edit the config file and your databases there is a GUI tool MySQL Workbench

Mac Ports Cheat Sheet

It’s worth reading the mac ports guide here, but here is my cut down cheat sheet of commands I actually use.

To install a port, e.g opencv

sudo port install opencv

To update ports definition list

sudo port selfupdate

Once, that’s finished you can update the outdated ports (this might take sometime depending on the number of updates)

sudo port upgrade outdated

or you can get a list of the outdated ports

port outdated

and upgraded them manually

sudo port upgrade opencv

This should also upgrade dependencies.

To get a list of installed ports

port installed

To search for a port, eg, PHP you could try

port search php

This will give you hundreds of port that mention PHP in their description. You can filter this, for example

port search --name --glob 'php*'

will return only the ports that start with php.

port info opencv

returns information about the opencv port (the lastest version, so watch out if your ports are out of date). A list of variants can be obtained with

port variants opencv

A port variant can be installed using the syntax:

sudo port install opencv +java

This installs the variant of opencv with java bindings. To see which variant is installed type:

port installed opencv

this lists the variant installed, plus the inactive versions of the library.

If a port fails to build it might be worth cleaning up first by running

sudo port clean opencv

and trying again.

To find out what and where port has installed files run

port contents opencv

To remove a port you use either of

sudo port uninstall opencv
sudo port uninstall --follow-dependents opencv

The second option removes the installed dependents as long as no other port is using them. If you don’t use the latter there will be a number of ports left on your system that you didn’t manually install. These are known as leaves and you can list or remove them with

port echo leaves
sudo port uninstall leaves

You might need to repeat the process since uninstalling leaves may create new leaves!

Port will also leave the outdated ports on your system. To remove them use

sudo port uninstall inactive