Using Multiprocessing shared memory with numpy matrices
One of the things one might need in scientific computing applications is to write a parallel computing software for computing some stuff, like matrices for example..
one of the famous parallel/distributed computing packages for python is the multiprocessing package, that has different functions for shared memory and message passing APIs between processes, out of the box you can use it to wite an application that runs more than one process sharing some variables and arrays, but in case of numpy arrays (matrices) that doesn't work!
I frankly don't know much on why doesn't it work, but it is said that this project can make it work, I didn't try it but I got out with a different solution that simply does the job for such a simple need I wanted.
This solution basically creates a shared basic python array (that multiprocessing package can normally deal with) and share it between processes
sharedArray = Array('d', n*n)
which will create an n*n 1-D array of decimals and we can treat it as an n*n 2-D matrix later..
after this we send it normally to the different processes we create
for i in range(P): processes.append(Process(target=compute, args=(i,P, sharedArray,))) processes[i].start()
Now we have each process main function "compute" got the shared array instance and can work with it normally,
But, you might want to deal with this shared array as a numpy matrix, so you can do this..
Matrix = np.frombuffer(sharedArray.get_obj()).reshape((n, n))
So you will have the object Matrix as a numpy array where you can use it with normal numpy array operators and this changes will be updated in the shared buffer...
But (Again!) because the equal operator replaces the whole buffer in cases like
Matrix = 1 * Matrix
you can't use the equal operator on the whole matrix as it will be not the shared buffer anymore, so the solution is to work on the shared matrix elementwise like
for i in range(n) for j in range(n) Matrix(i,j) = i+j
So you can create a process scoped result matrix with all the computations and then move it elementwise to the shared buffer, whatever suits you..
This will work and will update the shared buffer between all processes correctly and locking mechanisms should be working as well!