Distributing Tasks in the Net

This website describes a system to distribute computational-intensive tasks to autonomous nodes. It is client-server based and characterized by the following features:

• Support of heterogenous systems,
• Communication and data-exchange only by use of web technology,
• generic and usable for various applications,
• simple installation at the clients.

Task Distribution in homogenous LANs

Normally, jobs are distributed by the use of remote-access technologies: you log into a remote system (with SSH or similar protocols) and start the job. This is often automated by using various script-languages like python or perl. The input and output-files of the jobs need to be available on a common file system, NFS for example:

The problem is the following: The nodes are tightly coupled and it's hard to coordinate the single jobs, especially when there are many users and many computing-nodes. The used scripts have no knowledge about the load of the nodes and which user has started a run. Another disadvantege is that you need a homogeous LAN: Every node needs remote-access and must have the ability to mount a NFS-exported file-system ressource. In Unix-based systems ok, but in a firewall-protected internet, this is impractical. If we want to create a cluster-system for distributed coumputing which should be usable throughout the global internet, other technologies have to be used.

 

last update: 28.11.2007 by Matthias Bonn