parallelization and the master/slaves' env.io.atom_files_directory?
i'm separating a script that has worked into separate slave and master modules as described in http://salilab.org/modeller/manual/node435.html.
my master script looks like this:
> j = job() > j.append(local_slave()) > j.append(local_slave()) > > for afile in alignFiles: > > alnRunDir = '%s%s/' % (dataDir,aname) > if not os.path.isdir(alnRunDir): > print "doAllAlign: Creating directory",alnRunDir > os.makedirs(alnRunDir) > > for runIdx in range(nruns): > j.queue_task(SeedTask(argumentList)) > > for result in j.yield_tasks_unordered(): > print result
and my slave task like this:
> class SeedTask(task): > """A task to read in a PDB file on the slave, and return the resolution""" > > def run(self,dataDir,afile,aname,runIdx,randMethod,knownLbl,seqLbl,nmodPerRun,atomDeviation ): > env = environ() > env.io.atom_files_directory = [dataDir] > arg4seed = - (int(runIdx) + 2) > env = environ(arg4seed) > > runDir = '%s/%s/run_%02d/' % (dataDir,aname,runIdx) > if not os.path.isdir(runDir): > print "doAllAlign: Creating directory",runDir > os.makedirs(runDir) > > os.chdir(runDir) > > a = automodel(env, alnfile=afile, > knowns=knownLbl, sequence=seqLbl, > assess_methods=(assess.DOPE, assess.GA341)) > > a.starting_model = 1 > # 2do: test task / run ratio? > a.ending_model = nmodPerRun > > a.deviation = atomDeviation > if randMethod=='XYZ': > a.rand_method = randomize.xyz > elif randMethod=='DIHEDRAL': > a.rand_method = randomize.dihedrals > > a.make()
i'm stepping thru with a debugger, and it gets to the yield_tasks_unordered() generator in the master. but as i attempt to make the next step it throws an exception that seems to have to do with env.io.atom_files_directory now (in the master/slave version) not getting set correctly?
> Running 14 tasks on 2 slaves > /usr/bin/python2.6 /usr/lib/modeller9.10/bin/modslave.py -slave 127.0.1.1:59861:0:BGRESKQY >& seed_align_pp.slave0 > /usr/bin/python2.6 /usr/lib/modeller9.10/bin/modslave.py -slave 127.0.1.1:59861:1:ZCQMRSNQ >& seed_align_pp.slave1 > Identified slave <Slave on localhost> > Identified slave <Slave on localhost> > Traceback (most recent call last): > File "/home/rik/Data/pkg/eclipse/plugins/org.python.pydev.debug_2.2.3.2011100616/pysrc/pydevd.py", line 1267, in <module> > debugger.run(setup['file'], None, None) > File "/home/rik/Data/pkg/eclipse/plugins/org.python.pydev.debug_2.2.3.2011100616/pysrc/pydevd.py", line 1020, in run > pydev_imports.execfile(file, globals, locals) #execute the script > File "/home/rik/Code/eclipse/modeller/src/seed_align_pp.py", line 98, in <module> > doAllAlign() > File "/home/rik/Code/eclipse/modeller/src/seed_align_pp.py", line 91, in doAllAlign > for result in j.yield_tasks_unordered(): > File "/usr/lib/python2.6/dist-packages/modeller/parallel/job.py", line 146, in yield_tasks_unordered > for task in self._finish_all_tasks(): > File "/usr/lib/python2.6/dist-packages/modeller/parallel/job.py", line 164, in _finish_all_tasks > task = self._process_event(obj, s) > File "/usr/lib/python2.6/dist-packages/modeller/parallel/job.py", line 180, in _process_event > task = obj.task_results() > File "/usr/lib/python2.6/dist-packages/modeller/parallel/slave.py", line 61, in task_results > r = self.get_data(allow_heartbeat=True) > File "/usr/lib/python2.6/dist-packages/modeller/parallel/communicator.py", line 89, in get_data > (cmdtype, obj) = self._recv() > File "/usr/lib/python2.6/dist-packages/modeller/parallel/communicator.py", line 130, in _recv > raise RemoteError(obj.exc, self) > modeller.parallel.communicator.RemoteError: <type 'IOError'>: pdbnam_____E> Filename for PDB code not found: 1A04A.pdb > Directories: > Extensions : :.atm:.pdb:.ent:.crd > (Also tried prepending 'pdb', looking for .Z, .gz, .bz2, .7z, .xz, > and trying PDB-style subdirectories - e.g. ab for pdb1abc.ent) > from <Slave on localhost>
would you know what i'm doing wrong here? thanks again for your help.
Rik
On 11/01/2011 02:52 PM, R K Belew wrote: > my slave task like this: ... >> env = environ() >> env.io.atom_files_directory = [dataDir] >> arg4seed = - (int(runIdx) + 2) >> env = environ(arg4seed) ... > it throws an exception that seems > to have to do with env.io.atom_files_directory now (in the > master/slave version) not getting set correctly?
You've created two environ objects on your slave. Since you've assigned them to the same Python variable, the second will overwrite the first. What you probably want is
arg4seed = - (int(runIdx) + 2) env = environ(arg4seed) env.io.atom_files_directory = [dataDir]
Ben Webb, Modeller Caretaker
participants (2)
-
Modeller Caretaker
-
R K Belew