Piping numpy arrays to other processes in python

To pipe data from one process to another as a stream in python we need to pickle the object and pass it to the pipe stream.
In this example I’ve used Numpy arrays but this could be applied to any object that can be pickled in Python.
This took far too long to get working and I could find little information online on how to put it all together so here it is.
This code is Python 3 only, I’ve only run this on a Mac.

I’ve used binary as the stream rather than text purley becuase of effiencies. Numpy arrays can get huge! This means readline() is not
going to work. Instead, I send a single control byte , 1, for data and 0 for stop. This could be extended to include other control operations.
I then send the length of the data as a 8 byte int, followed by the data itself.

simpleSend.py

import numpy as np
import pickle
import sys
import io
import time

#define some control bytes
control_data=bytes([1])
control_stop=bytes([0])

def send_data(arr):
    dataStr=pickle.dumps(arr)  #pickle the data array into a byte array
    dlen=len(dataStr).to_bytes(8, byteorder='big') #find the length of the array and
    print(control_data.decode('latin-1'),end='',flush=True)  #convert this to a byte array
    print(dlen.decode('latin-1'), end='', flush=True)   #encode the data and write it
    print(dataStr.decode('latin-1'), end='', flush=True)  # end='' will remove that extra \r\n

def send_stop():
    print(control_stop.decode('latin-1'), end='', flush=True) 

#set the stdout such that it prints in latin-1,   sys.stdout.detach() is a binary stream
sys.stdout = io.TextIOWrapper(sys.stdout.detach(), encoding='latin-1')

for p in range(10):
    arr=np.ones((5000,500))*p  #generate some data
    send_data(arr)
    #the sleep is purely for testing and can be removed, ie does the reader fall over after a long delay
    time.sleep(.1)
send_stop()        

simpleReceiver.py

import numpy as np
import sys
import pickle

#define some control bytes
control_data=bytes([1])
control_stop=bytes([0])

while True:
    data=sys.stdin.buffer.read(1)   #read the control byte
    if data==control_data:
        data=sys.stdin.buffer.read(8)  #read the data length
        dlen=int.from_bytes(data, byteorder='big')
        print('data lenght %d'%dlen)        
        data=sys.stdin.buffer.read(dlen) #read the data        
        npd=pickle.loads(data)  #unpickle
        print(npd.shape)
        print(npd.max())
    elif data==control_stop:
        print('stopped')
        break
    else:
        print('Oh no')

to run this
python simpleSend.py | python simpleReceiver.py

If we want to use Python’s subprocess module to start simpleReceiver.py we basically need to write to the STDIN instead of print

import numpy as np
import pickle
import sys
import subprocess as sp

#define some control bytes
control_data=bytes([1])
control_stop=bytes([0])

def send_data(arr,buff):
    dataStr=pickle.dumps(arr)  #pickle the data array into a byte array
    dlen=len(dataStr).to_bytes(8, byteorder='big') #find the length of the array and
    mp.stdin.write(control_data)
    mp.stdin.write(dlen)
    mp.stdin.write(dataStr)
    mp.stdin.flush() #not sure this needed
     
def send_stop(mp):
    mp.stdin.write(control_stop)
    mp.stdin.flush()
     
try:
    mp = sp.Popen("python3 simpleReceiver.py",  shell = True,stdin=sp.PIPE)   
except sp.CalledProcessError as err:
    print('ERROR:', err)
    sys.exit(-1)

for p in range(10):
    arr=np.ones((5000,5000))*p  #generate some data
    send_data(arr,mp)
send_stop(mp)        

With such a large array 5000×5000 this takes sometime. Running it through the python profiler indicates about 75% of the time is taken by pickle.dumps and most of the rest of the remaining 25% is taken by the write operation. Numpy’s own method gives a speed increase. Replacing dataStr=pickle.dumps(arr) with dataStr=arr.tobytes() and npd=pickle.loads(data) with npd=np.frombuffer(data) more than halves the time taken but lose the shape and dtype information. This would have to be sent along with the data.

view raw
numpyPipe.md
hosted with ❤ by GitHub

Reskinning Max2play

Reskinning Max2play

Max2play is a great way to have an out of the box music server running on a Raspberry Pi with a touch screen. I’ve installed Max2play on a Raspberry Pi 3 and 7 inch touch screen. The only problem with it is that I would like to run other software on the Pi as well (such as control my Phillips Hue lighting). Out the box, the Pi now boots up and runs the full screen Jivelite app which controls the music.  There are two options I could think of: 1. Write a plugin for Jivelite to control the Hue lights, or 2, control the lighting with a separate app and have the ability to launch jivelite manually. I went for 2 since Jivelite is written in Lua which I don’t know.

The first problem is how to launch Jivelite manually. For this I added a Jivelite icon to the desktop.  I’m assuming Max2play is installed with Jivelite options and everything is working.

In the Jivelite plugin settings, disable autostart. In the Settings/Reboot tab, enable Autostart desktop (the desktop is normally started with the jivelite plugin so we need to enable it here).

In the Pi’s file manager go to Edit/Preferences – enable ‘Open files with Single click’ – it’s not easy double clicking with the touch screen. You might also want to increase the icon sizes whilst your there. These setting affect the desktop as well.

SSH into the Raspberry pi – either from the terminal or via the web plugin you can install. Go to the desktop folder and create a desktop launch shortcut. We will start by making one to launch jivelite. So create a text file with nano jivelite.desktop and add

[Desktop Entry]

Type=Application

Icon=/home/pi/music.png  <edit this to point to your own icon>

Name=Jivelite

Comment=Start the Jivelite music player

Exec=/opt/jivelite/jivelite/bin/jivelite

Save the file and that’s it. You should be now be able to launch jivelite from the desktop icon. The quit button does not quit the application (it seems to stop the music), so you have to quit from the menu options on Jivelite. (There appears to be a patch file that creates this behavior but I’ll investigate that at a latter date.)

To autostart at login, make a symbolic link to the .desktop file and place it in ~/.config/autostart

You may want to disable the screensaver which blanks the screen after 10 minutes as is. Remove the @xscreensaver -no-splash line from /etc/xdg/lxsession/LXDE/autostart and from ~/.config/lxsession/LXDE/autostart Stopping the screensaver server has no effect. I’m not sure what causes the screen to blank when running the GUI. It might be power saving in X. To disable the screensaver, the easiest method is to install a screensaver client and configure that not to run.

sudo apt-get install xscreensaver

After that’s completed there will be a screensaver option in the LXDE GUI menu. Run that and disable the screensaver from there.

 

vim cheat-sheet

A minimal cheat-sheet to get by in vim (vi /gvim)

There’s no doubt that vim/vi or gvim is an incredibly powerful editor. it’s also very lightweight and fast, making it an ideal editor on the Raspberry Pi. It does however have a learning curve. This in a minimal cheat-sheet of commands. The ones in bold are I think the ones you need to learn to be able to use it at a basic level. Also try running vimtutor from the command line for a tutorial.

Esc key – Normal mode

Inserting text

a -append

i – insert

o – open (inserts line below current line, O open above current line)

r – replace (replaces the character under the cursor with the next one entered,

R – replace mode (overwrite text until Esc pressed)

ce – change (deletes to end of word and then switches to insert mode) c$ deletes the rest of the line

Deleting text

x -delete character

dw – delete word      (d2w delete two words)

de – delete from cursor to end of word

d$ – delete from cursor to end of line

dd – delete the line  (2dd delete two lines)

Undo

u – undo

U – undo all changes on a line

Put, Past and Cut

p – put the contents of the buffer (paste) (this is the last thing deleted or yanked)

v – visual selections mode (highlights text for eg deletions or write to file with :w FILENAME)

y – yank (copies highlighted text), yw – yanks word

Moving about

gg – go to start of the file

GG – go to end of the file

504G – goto line 504

/  – search, n find next, N find previous

% – find matching bracket (,[,{

0 – move to start of the line, ^ move to first non space character.

$ – move to end of the line

Status

^G – file and position status

! – execute external command e.g. !ls

Writing and Reading

:w – save the file

:w FILENAME – save as FILENAME

:r FILENAME – inserts content of file here, you can also insert output of commands eg :r !ls

:q! -quit discarding changes

: x – quit saving changes (if there are any as opposed to :wq which always saves)

curl and sftp

For some reason curl does not come with sftp support built in some Linux distros (eg Mint). Type curl -V to see if its listed as a supported protocol. If it missing you need to download the source files from https://curl.haxx.se/download.html

Install the libssh development library which is libssh2-1-dev on my mint box. Un zip/tar the curl source and run from its directory

./configure

make

sudo make install

on the command line. When you run curl yo