Photo

Return to home page


E-mail

fredrik.sandin@gmail.com
(encryption key)

Official webpage

ltu.se/staff/f/fresan

Address

Fredrik Sandin,
EISLAB,
Luleå tekniska universitet,
971 87 Luleå,
Sweden.


Publications with Bibtex

Profile at Google Scholar


Distribution of calculations over SSH

Introduction

When starting the development of 3FCS as a Ph.D. student I had limited access to computing resources. I therefore decided to write a simple client-server application that could distribute calculations on the internet. The main design goals were that it should be simple to install on the computing nodes, and that no external libraries or administrator privileges should be required.

The solution I came up with is to use SSH and public-key authentication, so-called forced-command keys, and a client-server mechanism that allows clients to communicate with the standard input, output and error streams of the distributed processes in a transparent way. The requirements on a computing node are 1) that it is possible to connect to it via SSH from the server, 2) that it can compile/run the program that you want to distribute and 3) that a person with an account on that machine agrees to make the initial setup, which typically takes a few minutes. Because of the simple and generic philosophy behind the code it can be used in other contexts.

When submitting a job to a server, the client receives a unique identifier that can be used to poll messages and the result from the server. Clients can also make blocking submissions, which do not return until the result has been calculated. In principle, any program that communicates via stdin, stdout and stderr can be adapted to communicate in this distributed environment.

Example

Here is an example of how a client sends a job to a server and then prints the result on the screen:

from client import *
	
job = Job()
job.create('server.address.org', PORTNUMBER, "what is the meaning of life?", block=True)

print 'Job ID:', job.job_id()
print 'Result sent to stdout:', job.result()
print 'Messages sent to stderr:', job.messages()
	

In this example the client has to know the address of the server, but nothing about the computers that the server communicates with. The server will transparently send the job to an idle computer and then return the result when it is available. If the job is created with 'block=False' the function call will return immediately. In that case the client has to poll the result from the server. The benefit of this is that the client can do other things while waiting for the result, e.g., continuously polling and displaying messages from the program.

A client can create multiple jobs, which then will be distributed by the server. This is how a large calculation is divided into parts that can be completed in shorter time. For more information, see the documentation and the examples in the source code package.

Download

You can download the Python source code here, or via this direct link.

License and terms of use

This is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.


Software

3FCS
ASOUND
CBVS
Dircheck
Etherpad Latex scripts (github)
Femtolensing calculator
Random indexing
Shellmap
SSH-distributed calculations

Technical reports

TDP derivatives (extends Appendix E/F)
Fault detection in district energy systems
Intelligent Industrial Processes

Bulletin board

Thu Oct 08, 2015
Master thesis proposal on embedded streaming text analytics.

Wed Oct 07, 2015
Gunnar Öquist fellows, vice-chancellors and representatives from research funding organisations in Sweden were invited to IVA by the Kempe Foundations to present their view on and discuss the fellowship programme. A very interesting and stimulating meeting.

Thu Sep 24, 2015
Master thesis proposal on learning condition monitoring system.

Mon Sep 14, 2015
Dr. Vasaki Ponnusamy from Quest International University in Malaysia is visiting us for one month with support from STINT.

Mon Aug 31, 2015
Sergio visits EUSIPCO 2015 and presents our paper.

Wed May 27, 2015
Sergio Martin del Campo Barraza presents his licentiate thesis on autonomous sensor systems. Congratulations! Thanks to the opponent, Karl Skretting, for his helpful questions at the seminar.

Mon Apr 27, 2015
Participated at CapoCaccia, great event! Implemented a realtime Matlab / Mex interface (screenshot) for the DAVIS device and learned more about criticality and the ROLLS.

Fri Dec 12, 2014
Gunnar Öquist Fellow ceremony. A memorable day and a great and timely opportunity. More info here.

Thu Nov 13, 2014
Received an encouraging letter from the student union. Thank you for nominating me!

Fri Nov 7, 2014
Presenting our work on interoperability by learning at BICA 2014 in Boston.

Wed Oct 1, 2014
Blerim defends his Ph.D. thesis. Congrats! See also the news article.

Thu Jun 5, 2014
Workshop on bio-inspired computation, organized at UTP, Malaysia. Thanks to STINT, UTP and MMU for support.

Fri Apr 25, 2014
Ph.D. student Sergio Martin del Campo attends MLSS2014 and presents a poster.

News archive.


Valid XHTML 1.0 Transitional