we use cross-linking data as simple distance restraints - nothing fancy :)

On Dec 9, 2008, at 4:18 PM, Andrej Sali wrote:

> i trust that the work on the cross-linking restraints code in IMP is > based on the generalized description / classification of such > restraints in the recent NCDIR proposal? > > thanks, andrej > > -- > Andrej Sali, Ph.D. > Professor and Vice Chair, Department of Biopharmaceutical Sciences > Department of Pharmaceutical Chemistry > California Institute for Quantitative Biosciences > University of California at San Francisco > UCSF MC 2552 > Byers Hall Room 503B > 1700 4th Street > San Francisco, CA 94158-2330, USA > Tel +1 (415) 514-4227; Fax +1 (415) 514-4231 > Assistant, Ms. Karin Asensio, Tel +1 (415)514-4228; Lab +1 (415) > 514-4232, 4233, 4258 > Email sali@salilab.org; Web http://salilab.org > > > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev

In general, I think having queries on the whole collection of particles in the model is not a good idea (since other people's code, restraints or states can add particles to the model and you can never be sure what those look like).

There is already functionality to search a Hierarchy (although it is more aimed at C++-- we could use a python interface which takes takes a python lambda function to make it more convenient to use in python). And python has all sorts of features for searching a list (and C++ has a few). It is not clear to me that we could provide an interface that is general and much more concise.

As a slight simplification for python users, we could provide a function which takes a list of key, value pairs (with keys of arbitrary type) and a container. It is a bit messier to provide this interface in C++ as we would have to have a separate list per type.

Another thing to simplify such search would be a "DefaultValuesDecorator" which wraps a particle and pretend it has all attributes, just providing default values for missing ones. This would obviate the need to check for an attribute before matching against it.

What sort of queries do you all do?

On Dec 9, 2008, at 8:43 AM, Dina Schneidman wrote:

> Hi, > > I need this too (surprisingly). Usually I do it with mapping between > the particle and the attribute. > It is simple. however it is unclear where should we put such a > mapping. Putting it in a model > could be the best, however not everyone needs it. So it means > somewhere else or extending the Model to ProteinModel? > > Dina > > P.S. skype me, we can talk about it > > > On Tue, Dec 9, 2008 at 7:03 AM, Keren Lasker kerenl@salilab.org > wrote: >> hi all, >> >> Frido and I find ourselves many times need to query particles based >> on >> attribute values. >> Few such examples: a protein with a specific name, particles with a >> specific >> residue range. >> >> I think that it would be very useful to have something similar to >> SQL >> queries on the particles DB. >> Bret might had something similar implemented - but it is probably >> obsolete. >> IMP.Atom will probably need such functionality as well. >> >> has anyone took a look at that before ? >> >> thank you, >> Keren. >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev@salilab.org >> https://salilab.org/mailman/listinfo/imp-dev >> > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev

On Dec 9, 2008, at 9:29 AM, Keren Lasker wrote:

> For the 26S project we currently do: > get particle by name How could we beat something like: [x for x in myparticles if x.has_attribute(name) and x.get_value(name) == "myname"] with an SQL query? > > get a set of particles within a residue number range again, some variant on: [x for x in molecular_hierarchy_get_by_type(root, MolecularHierarchyDecorator.RESIDUE) if ResidueDecorator(x).get_index() > lb and ResidueDecorator(x).get_index() <ub]

or C++, something like BOOST_FOREACH(Particle *p, molecular_hierarchy_get_by_type(root, MolecularHierarchyDecorator.RESIDUE)) { if (ResidueDecorator(p).get_index() > lb and ResidueDecorator(p).get_index() <ub) { // do something } } > > > > the solution of a function that gets (key,value,container) seems > like a good solution. > However - it can be more complicated : > 1. it can interact with the hierarchy - give me the residues range > within this protein for example - so we should probably also allow > for a hierarchy handle in the interface. > 2. we might want to ask residue range + some other property such as > have structural coverage or do not. Therefore I think that a sql > type string can be more general than a list of attributes - because > you do not know how they are related. But the added complication is why I would suggest sticking with C++ or python. Lambda functions or list comprehensions support very general logic (more so than SQL) and allow you to leverage existing code. SQL would make it really hard to use any of the existing functionality and require lots of things be exposed in another language. For example, try to find all particles close to a point in SQL? It is kind of ugly.

> > > > On Tue, 9 Dec 2008, Daniel Russel wrote: > >> In general, I think having queries on the whole collection of >> particles in the model is not a good idea (since other people's >> code, restraints or states can add particles to the model and you >> can never be sure what those look like). >> >> There is already functionality to search a Hierarchy (although it >> is more aimed at C++-- we could use a python interface which takes >> takes a python lambda function to make it more convenient to use in >> python). And python has all sorts of features for searching a list >> (and C++ has a few). >> It is not clear to me that we could provide an interface that is >> general and much more concise. >> >> As a slight simplification for python users, we could provide a >> function which takes a list of key, value pairs (with keys of >> arbitrary type) and a container. It is a bit messier to provide >> this interface in C++ as we would have to have a separate list per >> type. >> >> Another thing to simplify such search would be a >> "DefaultValuesDecorator" which wraps a particle and pretend it has >> all attributes, just providing default values for missing ones. >> This would obviate the need to check for an attribute before >> matching against it. >> >> What sort of queries do you all do? >> >> >> >> On Dec 9, 2008, at 8:43 AM, Dina Schneidman wrote: >> >>> Hi, >>> I need this too (surprisingly). Usually I do it with mapping between >>> the particle and the attribute. >>> It is simple. however it is unclear where should we put such a >>> mapping. Putting it in a model >>> could be the best, however not everyone needs it. So it means >>> somewhere else or extending the Model to ProteinModel? >>> Dina >>> P.S. skype me, we can talk about it >>> On Tue, Dec 9, 2008 at 7:03 AM, Keren Lasker kerenl@salilab.org >>> wrote: >>>> hi all, >>>> Frido and I find ourselves many times need to query particles >>>> based on >>>> attribute values. >>>> Few such examples: a protein with a specific name, particles with >>>> a specific >>>> residue range. >>>> I think that it would be very useful to have something similar >>>> to SQL >>>> queries on the particles DB. >>>> Bret might had something similar implemented - but it is >>>> probably obsolete. >>>> IMP.Atom will probably need such functionality as well. >>>> has anyone took a look at that before ? >>>> thank you, >>>> Keren. >>>> _______________________________________________ >>>> IMP-dev mailing list >>>> IMP-dev@salilab.org >>>> https://salilab.org/mailman/listinfo/imp-dev >>> _______________________________________________ >>> IMP-dev mailing list >>> IMP-dev@salilab.org >>> https://salilab.org/mailman/listinfo/imp-dev >> >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev@salilab.org >> https://salilab.org/mailman/listinfo/imp-dev > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev

On Dec 9, 2008, at 9:55 AM, Keren Lasker wrote:

> Daniel - we all know how to run for loops ;) > I just thought it make sense to have something more efficient :) Ahhh, I thought you were aiming for conciseness :-) My bad. Trying to pay attention to too many things at once...

How about a ScoreState which builds an index using an arbitrary tuple of IntKeys (or an arbitrary tuple of keys). It would be very trivial to write (using templates in C++, perhaps an alternate python version would be good).

> > > On Tue, 9 Dec 2008, Daniel Russel wrote: > >> >> On Dec 9, 2008, at 9:29 AM, Keren Lasker wrote: >> >>> For the 26S project we currently do: >>> get particle by name >> How could we beat something like: >> [x for x in myparticles if x.has_attribute(name) and >> x.get_value(name) == "myname"] >> with an SQL query? >>> get a set of particles within a residue number range >> again, some variant on: >> [x for x in molecular_hierarchy_get_by_type(root, >> MolecularHierarchyDecorator.RESIDUE) if >> ResidueDecorator(x).get_index() > lb and >> ResidueDecorator(x).get_index() <ub] >> >> or C++, something like >> BOOST_FOREACH(Particle *p, molecular_hierarchy_get_by_type(root, >> MolecularHierarchyDecorator.RESIDUE)) { >> if (ResidueDecorator(p).get_index() > lb and >> ResidueDecorator(p).get_index() <ub) { >> // do something >> } >> } >>> the solution of a function that gets (key,value,container) seems >>> like a good solution. >>> However - it can be more complicated : >>> 1. it can interact with the hierarchy - give me the residues >>> range within this protein for example - so we should probably also >>> allow for a hierarchy handle in the interface. >>> 2. we might want to ask residue range + some other property such >>> as have structural coverage or do not. Therefore I think that a >>> sql type string can be more general than a list of attributes - >>> because you do not know how they are related. >> But the added complication is why I would suggest sticking with C++ >> or python. Lambda functions or list comprehensions support very >> general logic (more so than SQL) and allow you to leverage existing >> code. SQL would make it really hard to use any of the existing >> functionality and require lots of things be exposed in another >> language. For example, try to find all particles close to a point >> in SQL? It is kind of ugly. >> >>> On Tue, 9 Dec 2008, Daniel Russel wrote: >>>> In general, I think having queries on the whole collection of >>>> particles in the model is not a good idea (since other people's >>>> code, restraints or states can add particles to the model and you >>>> can never be sure what those look like). >>>> There is already functionality to search a Hierarchy (although it >>>> is more aimed at C++-- we could use a python interface which >>>> takes takes a python lambda function to make it more convenient >>>> to use in python). And python has all sorts of features for >>>> searching a list (and C++ has a few). >>>> It is not clear to me that we could provide an interface that is >>>> general and much more concise. >>>> As a slight simplification for python users, we could provide a >>>> function which takes a list of key, value pairs (with keys of >>>> arbitrary type) and a container. It is a bit messier to provide >>>> this interface in C++ as we would have to have a separate list >>>> per type. >>>> Another thing to simplify such search would be a >>>> "DefaultValuesDecorator" which wraps a particle and pretend it >>>> has all attributes, just providing default values for missing >>>> ones. This would obviate the need to check for an attribute >>>> before matching against it. >>>> What sort of queries do you all do? >>>> On Dec 9, 2008, at 8:43 AM, Dina Schneidman wrote: >>>>> Hi, >>>>> I need this too (surprisingly). Usually I do it with mapping >>>>> between >>>>> the particle and the attribute. >>>>> It is simple. however it is unclear where should we put such a >>>>> mapping. Putting it in a model >>>>> could be the best, however not everyone needs it. So it means >>>>> somewhere else or extending the Model to ProteinModel? >>>>> Dina >>>>> P.S. skype me, we can talk about it >>>>> On Tue, Dec 9, 2008 at 7:03 AM, Keren Lasker >>>>> kerenl@salilab.org wrote: >>>>>> hi all, >>>>>> Frido and I find ourselves many times need to query particles >>>>>> based on >>>>>> attribute values. >>>>>> Few such examples: a protein with a specific name, particles >>>>>> with a specific >>>>>> residue range. >>>>>> I think that it would be very useful to have something similar >>>>>> to SQL >>>>>> queries on the particles DB. >>>>>> Bret might had something similar implemented - but it is >>>>>> probably obsolete. >>>>>> IMP.Atom will probably need such functionality as well. >>>>>> has anyone took a look at that before ? >>>>>> thank you, >>>>>> Keren. >>>>>> _______________________________________________ >>>>>> IMP-dev mailing list >>>>>> IMP-dev@salilab.org >>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>> _______________________________________________ >>>>> IMP-dev mailing list >>>>> IMP-dev@salilab.org >>>>> https://salilab.org/mailman/listinfo/imp-dev >>>> _______________________________________________ >>>> IMP-dev mailing list >>>> IMP-dev@salilab.org >>>> https://salilab.org/mailman/listinfo/imp-dev >>> _______________________________________________ >>> IMP-dev mailing list >>> IMP-dev@salilab.org >>> https://salilab.org/mailman/listinfo/imp-dev >> >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev@salilab.org >> https://salilab.org/mailman/listinfo/imp-dev > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev

maybe it's a simple solution in order to have it in a model, but conceptually this indexing has nothing to do with scoring

On Tue, Dec 9, 2008 at 10:42 AM, Daniel Russel drussel@gmail.com wrote: > From what I understand, what you want is a way of specifying what indexes we > want build (not a way of specifying queries). We could easily provide > ScoreStates for indexes based on: > - set of discrete valued attributes > - D-dimensional interval queries on float values > > > On Dec 9, 2008, at 9:55 AM, Keren Lasker wrote: > >> Daniel - we all know how to run for loops ;) >> I just thought it make sense to have something more efficient :) >> >> On Tue, 9 Dec 2008, Daniel Russel wrote: >> >>> >>> On Dec 9, 2008, at 9:29 AM, Keren Lasker wrote: >>> >>>> For the 26S project we currently do: >>>> get particle by name >>> >>> How could we beat something like: >>> [x for x in myparticles if x.has_attribute(name) and x.get_value(name) == >>> "myname"] >>> with an SQL query? >>>> >>>> get a set of particles within a residue number range >>> >>> again, some variant on: >>> [x for x in molecular_hierarchy_get_by_type(root, >>> MolecularHierarchyDecorator.RESIDUE) if >>> ResidueDecorator(x).get_index() > lb and ResidueDecorator(x).get_index() >>> <ub] >>> >>> or C++, something like >>> BOOST_FOREACH(Particle *p, molecular_hierarchy_get_by_type(root, >>> MolecularHierarchyDecorator.RESIDUE)) { >>> if (ResidueDecorator(p).get_index() > lb and >>> ResidueDecorator(p).get_index() <ub) { >>> // do something >>> } >>> } >>>> >>>> the solution of a function that gets (key,value,container) seems like a >>>> good solution. >>>> However - it can be more complicated : >>>> 1. it can interact with the hierarchy - give me the residues range >>>> within this protein for example - so we should probably also allow for a >>>> hierarchy handle in the interface. >>>> 2. we might want to ask residue range + some other property such as have >>>> structural coverage or do not. Therefore I think that a sql type string can >>>> be more general than a list of attributes - because you do not know how they >>>> are related. >>> >>> But the added complication is why I would suggest sticking with C++ or >>> python. Lambda functions or list comprehensions support very general logic >>> (more so than SQL) and allow you to leverage existing code. SQL would make >>> it really hard to use any of the existing functionality and require lots of >>> things be exposed in another language. For example, try to find all >>> particles close to a point in SQL? It is kind of ugly. >>> >>>> On Tue, 9 Dec 2008, Daniel Russel wrote: >>>>> >>>>> In general, I think having queries on the whole collection of particles >>>>> in the model is not a good idea (since other people's code, restraints or >>>>> states can add particles to the model and you can never be sure what those >>>>> look like). >>>>> There is already functionality to search a Hierarchy (although it is >>>>> more aimed at C++-- we could use a python interface which takes takes a >>>>> python lambda function to make it more convenient to use in python). And >>>>> python has all sorts of features for searching a list (and C++ has a few). >>>>> It is not clear to me that we could provide an interface that is >>>>> general and much more concise. >>>>> As a slight simplification for python users, we could provide a >>>>> function which takes a list of key, value pairs (with keys of arbitrary >>>>> type) and a container. It is a bit messier to provide this interface in C++ >>>>> as we would have to have a separate list per type. >>>>> Another thing to simplify such search would be a >>>>> "DefaultValuesDecorator" which wraps a particle and pretend it has all >>>>> attributes, just providing default values for missing ones. This would >>>>> obviate the need to check for an attribute before matching against it. >>>>> What sort of queries do you all do? >>>>> On Dec 9, 2008, at 8:43 AM, Dina Schneidman wrote: >>>>>> >>>>>> Hi, >>>>>> I need this too (surprisingly). Usually I do it with mapping between >>>>>> the particle and the attribute. >>>>>> It is simple. however it is unclear where should we put such a >>>>>> mapping. Putting it in a model >>>>>> could be the best, however not everyone needs it. So it means >>>>>> somewhere else or extending the Model to ProteinModel? >>>>>> Dina >>>>>> P.S. skype me, we can talk about it >>>>>> On Tue, Dec 9, 2008 at 7:03 AM, Keren Lasker kerenl@salilab.org >>>>>> wrote: >>>>>>> >>>>>>> hi all, >>>>>>> Frido and I find ourselves many times need to query particles based >>>>>>> on >>>>>>> attribute values. >>>>>>> Few such examples: a protein with a specific name, particles with a >>>>>>> specific >>>>>>> residue range. >>>>>>> I think that it would be very useful to have something similar to >>>>>>> SQL >>>>>>> queries on the particles DB. >>>>>>> Bret might had something similar implemented - but it is probably >>>>>>> obsolete. >>>>>>> IMP.Atom will probably need such functionality as well. >>>>>>> has anyone took a look at that before ? >>>>>>> thank you, >>>>>>> Keren. >>>>>>> _______________________________________________ >>>>>>> IMP-dev mailing list >>>>>>> IMP-dev@salilab.org >>>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>>> >>>>>> _______________________________________________ >>>>>> IMP-dev mailing list >>>>>> IMP-dev@salilab.org >>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>> >>>>> _______________________________________________ >>>>> IMP-dev mailing list >>>>> IMP-dev@salilab.org >>>>> https://salilab.org/mailman/listinfo/imp-dev >>>> >>>> _______________________________________________ >>>> IMP-dev mailing list >>>> IMP-dev@salilab.org >>>> https://salilab.org/mailman/listinfo/imp-dev >>> >>> _______________________________________________ >>> IMP-dev mailing list >>> IMP-dev@salilab.org >>> https://salilab.org/mailman/listinfo/imp-dev >> >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev@salilab.org >> https://salilab.org/mailman/listinfo/imp-dev > > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev >

but you don't always want to update you indexing before/after scoring

On Tue, Dec 9, 2008 at 11:06 AM, Daniel Russel drussel@gmail.com wrote: > Score states don't have anything to do with scoring either :-) they are just > updated before scoring since that is when things can change during > optimization. They used to just be called States which is perhaps clearer. > > > > On Dec 9, 2008, at 10:56 AM, "Dina Schneidman" duhovka@gmail.com wrote: > >> maybe it's a simple solution in order to have it in a model, but >> conceptually this indexing has nothing to do with scoring >> >> On Tue, Dec 9, 2008 at 10:42 AM, Daniel Russel drussel@gmail.com wrote: >>> >>> From what I understand, what you want is a way of specifying what indexes >>> we >>> want build (not a way of specifying queries). We could easily provide >>> ScoreStates for indexes based on: >>> - set of discrete valued attributes >>> - D-dimensional interval queries on float values >>> >>> >>> On Dec 9, 2008, at 9:55 AM, Keren Lasker wrote: >>> >>>> Daniel - we all know how to run for loops ;) >>>> I just thought it make sense to have something more efficient :) >>>> >>>> On Tue, 9 Dec 2008, Daniel Russel wrote: >>>> >>>>> >>>>> On Dec 9, 2008, at 9:29 AM, Keren Lasker wrote: >>>>> >>>>>> For the 26S project we currently do: >>>>>> get particle by name >>>>> >>>>> How could we beat something like: >>>>> [x for x in myparticles if x.has_attribute(name) and x.get_value(name) >>>>> == >>>>> "myname"] >>>>> with an SQL query? >>>>>> >>>>>> get a set of particles within a residue number range >>>>> >>>>> again, some variant on: >>>>> [x for x in molecular_hierarchy_get_by_type(root, >>>>> MolecularHierarchyDecorator.RESIDUE) if >>>>> ResidueDecorator(x).get_index() > lb and >>>>> ResidueDecorator(x).get_index() >>>>> <ub] >>>>> >>>>> or C++, something like >>>>> BOOST_FOREACH(Particle *p, molecular_hierarchy_get_by_type(root, >>>>> MolecularHierarchyDecorator.RESIDUE)) { >>>>> if (ResidueDecorator(p).get_index() > lb and >>>>> ResidueDecorator(p).get_index() <ub) { >>>>> // do something >>>>> } >>>>> } >>>>>> >>>>>> the solution of a function that gets (key,value,container) seems like >>>>>> a >>>>>> good solution. >>>>>> However - it can be more complicated : >>>>>> 1. it can interact with the hierarchy - give me the residues range >>>>>> within this protein for example - so we should probably also allow for >>>>>> a >>>>>> hierarchy handle in the interface. >>>>>> 2. we might want to ask residue range + some other property such as >>>>>> have >>>>>> structural coverage or do not. Therefore I think that a sql type >>>>>> string can >>>>>> be more general than a list of attributes - because you do not know >>>>>> how they >>>>>> are related. >>>>> >>>>> But the added complication is why I would suggest sticking with C++ or >>>>> python. Lambda functions or list comprehensions support very general >>>>> logic >>>>> (more so than SQL) and allow you to leverage existing code. SQL would >>>>> make >>>>> it really hard to use any of the existing functionality and require >>>>> lots of >>>>> things be exposed in another language. For example, try to find all >>>>> particles close to a point in SQL? It is kind of ugly. >>>>> >>>>>> On Tue, 9 Dec 2008, Daniel Russel wrote: >>>>>>> >>>>>>> In general, I think having queries on the whole collection of >>>>>>> particles >>>>>>> in the model is not a good idea (since other people's code, >>>>>>> restraints or >>>>>>> states can add particles to the model and you can never be sure what >>>>>>> those >>>>>>> look like). >>>>>>> There is already functionality to search a Hierarchy (although it is >>>>>>> more aimed at C++-- we could use a python interface which takes takes >>>>>>> a >>>>>>> python lambda function to make it more convenient to use in python). >>>>>>> And >>>>>>> python has all sorts of features for searching a list (and C++ has a >>>>>>> few). >>>>>>> It is not clear to me that we could provide an interface that is >>>>>>> general and much more concise. >>>>>>> As a slight simplification for python users, we could provide a >>>>>>> function which takes a list of key, value pairs (with keys of >>>>>>> arbitrary >>>>>>> type) and a container. It is a bit messier to provide this interface >>>>>>> in C++ >>>>>>> as we would have to have a separate list per type. >>>>>>> Another thing to simplify such search would be a >>>>>>> "DefaultValuesDecorator" which wraps a particle and pretend it has >>>>>>> all >>>>>>> attributes, just providing default values for missing ones. This >>>>>>> would >>>>>>> obviate the need to check for an attribute before matching against >>>>>>> it. >>>>>>> What sort of queries do you all do? >>>>>>> On Dec 9, 2008, at 8:43 AM, Dina Schneidman wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> I need this too (surprisingly). Usually I do it with mapping between >>>>>>>> the particle and the attribute. >>>>>>>> It is simple. however it is unclear where should we put such a >>>>>>>> mapping. Putting it in a model >>>>>>>> could be the best, however not everyone needs it. So it means >>>>>>>> somewhere else or extending the Model to ProteinModel? >>>>>>>> Dina >>>>>>>> P.S. skype me, we can talk about it >>>>>>>> On Tue, Dec 9, 2008 at 7:03 AM, Keren Lasker kerenl@salilab.org >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> hi all, >>>>>>>>> Frido and I find ourselves many times need to query particles based >>>>>>>>> on >>>>>>>>> attribute values. >>>>>>>>> Few such examples: a protein with a specific name, particles with a >>>>>>>>> specific >>>>>>>>> residue range. >>>>>>>>> I think that it would be very useful to have something similar to >>>>>>>>> SQL >>>>>>>>> queries on the particles DB. >>>>>>>>> Bret might had something similar implemented - but it is probably >>>>>>>>> obsolete. >>>>>>>>> IMP.Atom will probably need such functionality as well. >>>>>>>>> has anyone took a look at that before ? >>>>>>>>> thank you, >>>>>>>>> Keren. >>>>>>>>> _______________________________________________ >>>>>>>>> IMP-dev mailing list >>>>>>>>> IMP-dev@salilab.org >>>>>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> IMP-dev mailing list >>>>>>>> IMP-dev@salilab.org >>>>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>>>> >>>>>>> _______________________________________________ >>>>>>> IMP-dev mailing list >>>>>>> IMP-dev@salilab.org >>>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>>> >>>>>> _______________________________________________ >>>>>> IMP-dev mailing list >>>>>> IMP-dev@salilab.org >>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>> >>>>> _______________________________________________ >>>>> IMP-dev mailing list >>>>> IMP-dev@salilab.org >>>>> https://salilab.org/mailman/listinfo/imp-dev >>>> >>>> _______________________________________________ >>>> IMP-dev mailing list >>>> IMP-dev@salilab.org >>>> https://salilab.org/mailman/listinfo/imp-dev >>> >>> _______________________________________________ >>> IMP-dev mailing list >>> IMP-dev@salilab.org >>> https://salilab.org/mailman/listinfo/imp-dev >>> >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev@salilab.org >> https://salilab.org/mailman/listinfo/imp-dev > > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev >

Daniel - I am sorry I was a way for a few hours - but in the end I did not really follow from this thread what type of queries would be possible.

On Tue, 9 Dec 2008, Daniel Russel wrote:

> True, but is our only current hook for keeping things up to date. One > alternative would be to add hooks to monitor attribute changes, but that > would be tricky to make not expensive. You don't have to add states to a > model though, so you could update it manually if you want. > > > > On Dec 9, 2008, at 11:08 AM, "Dina Schneidman" duhovka@gmail.com wrote: > >> but you don't always want to update you indexing before/after scoring >> >> On Tue, Dec 9, 2008 at 11:06 AM, Daniel Russel drussel@gmail.com wrote: >>> Score states don't have anything to do with scoring either :-) they are >>> just >>> updated before scoring since that is when things can change during >>> optimization. They used to just be called States which is perhaps >>> clearer. >>> >>> >>> >>> On Dec 9, 2008, at 10:56 AM, "Dina Schneidman" duhovka@gmail.com >>> wrote: >>> >>>> maybe it's a simple solution in order to have it in a model, but >>>> conceptually this indexing has nothing to do with scoring >>>> >>>> On Tue, Dec 9, 2008 at 10:42 AM, Daniel Russel drussel@gmail.com >>>> wrote: >>>>> >>>>> From what I understand, what you want is a way of specifying what >>>>> indexes >>>>> we >>>>> want build (not a way of specifying queries). We could easily provide >>>>> ScoreStates for indexes based on: >>>>> - set of discrete valued attributes >>>>> - D-dimensional interval queries on float values >>>>> >>>>> >>>>> On Dec 9, 2008, at 9:55 AM, Keren Lasker wrote: >>>>> >>>>>> Daniel - we all know how to run for loops ;) >>>>>> I just thought it make sense to have something more efficient :) >>>>>> >>>>>> On Tue, 9 Dec 2008, Daniel Russel wrote: >>>>>> >>>>>>> >>>>>>> On Dec 9, 2008, at 9:29 AM, Keren Lasker wrote: >>>>>>> >>>>>>>> For the 26S project we currently do: >>>>>>>> get particle by name >>>>>>> >>>>>>> How could we beat something like: >>>>>>> [x for x in myparticles if x.has_attribute(name) and >>>>>>> x.get_value(name) >>>>>>> == >>>>>>> "myname"] >>>>>>> with an SQL query? >>>>>>>> >>>>>>>> get a set of particles within a residue number range >>>>>>> >>>>>>> again, some variant on: >>>>>>> [x for x in molecular_hierarchy_get_by_type(root, >>>>>>> MolecularHierarchyDecorator.RESIDUE) if >>>>>>> ResidueDecorator(x).get_index() > lb and >>>>>>> ResidueDecorator(x).get_index() >>>>>>> <ub] >>>>>>> >>>>>>> or C++, something like >>>>>>> BOOST_FOREACH(Particle *p, molecular_hierarchy_get_by_type(root, >>>>>>> MolecularHierarchyDecorator.RESIDUE)) { >>>>>>> if (ResidueDecorator(p).get_index() > lb and >>>>>>> ResidueDecorator(p).get_index() <ub) { >>>>>>> // do something >>>>>>> } >>>>>>> } >>>>>>>> >>>>>>>> the solution of a function that gets (key,value,container) seems >>>>>>>> like >>>>>>>> a >>>>>>>> good solution. >>>>>>>> However - it can be more complicated : >>>>>>>> 1. it can interact with the hierarchy - give me the residues range >>>>>>>> within this protein for example - so we should probably also allow >>>>>>>> for >>>>>>>> a >>>>>>>> hierarchy handle in the interface. >>>>>>>> 2. we might want to ask residue range + some other property such as >>>>>>>> have >>>>>>>> structural coverage or do not. Therefore I think that a sql type >>>>>>>> string can >>>>>>>> be more general than a list of attributes - because you do not know >>>>>>>> how they >>>>>>>> are related. >>>>>>> >>>>>>> But the added complication is why I would suggest sticking with C++ >>>>>>> or >>>>>>> python. Lambda functions or list comprehensions support very general >>>>>>> logic >>>>>>> (more so than SQL) and allow you to leverage existing code. SQL >>>>>>> would >>>>>>> make >>>>>>> it really hard to use any of the existing functionality and require >>>>>>> lots of >>>>>>> things be exposed in another language. For example, try to find all >>>>>>> particles close to a point in SQL? It is kind of ugly. >>>>>>> >>>>>>>> On Tue, 9 Dec 2008, Daniel Russel wrote: >>>>>>>>> >>>>>>>>> In general, I think having queries on the whole collection of >>>>>>>>> particles >>>>>>>>> in the model is not a good idea (since other people's code, >>>>>>>>> restraints or >>>>>>>>> states can add particles to the model and you can never be sure >>>>>>>>> what >>>>>>>>> those >>>>>>>>> look like). >>>>>>>>> There is already functionality to search a Hierarchy (although it >>>>>>>>> is >>>>>>>>> more aimed at C++-- we could use a python interface which takes >>>>>>>>> takes >>>>>>>>> a >>>>>>>>> python lambda function to make it more convenient to use in >>>>>>>>> python). >>>>>>>>> And >>>>>>>>> python has all sorts of features for searching a list (and C++ has >>>>>>>>> a >>>>>>>>> few). >>>>>>>>> It is not clear to me that we could provide an interface that is >>>>>>>>> general and much more concise. >>>>>>>>> As a slight simplification for python users, we could provide a >>>>>>>>> function which takes a list of key, value pairs (with keys of >>>>>>>>> arbitrary >>>>>>>>> type) and a container. It is a bit messier to provide this >>>>>>>>> interface >>>>>>>>> in C++ >>>>>>>>> as we would have to have a separate list per type. >>>>>>>>> Another thing to simplify such search would be a >>>>>>>>> "DefaultValuesDecorator" which wraps a particle and pretend it has >>>>>>>>> all >>>>>>>>> attributes, just providing default values for missing ones. This >>>>>>>>> would >>>>>>>>> obviate the need to check for an attribute before matching against >>>>>>>>> it. >>>>>>>>> What sort of queries do you all do? >>>>>>>>> On Dec 9, 2008, at 8:43 AM, Dina Schneidman wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> I need this too (surprisingly). Usually I do it with mapping >>>>>>>>>> between >>>>>>>>>> the particle and the attribute. >>>>>>>>>> It is simple. however it is unclear where should we put such a >>>>>>>>>> mapping. Putting it in a model >>>>>>>>>> could be the best, however not everyone needs it. So it means >>>>>>>>>> somewhere else or extending the Model to ProteinModel? >>>>>>>>>> Dina >>>>>>>>>> P.S. skype me, we can talk about it >>>>>>>>>> On Tue, Dec 9, 2008 at 7:03 AM, Keren Lasker kerenl@salilab.org >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> hi all, >>>>>>>>>>> Frido and I find ourselves many times need to query particles >>>>>>>>>>> based >>>>>>>>>>> on >>>>>>>>>>> attribute values. >>>>>>>>>>> Few such examples: a protein with a specific name, particles >>>>>>>>>>> with a >>>>>>>>>>> specific >>>>>>>>>>> residue range. >>>>>>>>>>> I think that it would be very useful to have something similar >>>>>>>>>>> to >>>>>>>>>>> SQL >>>>>>>>>>> queries on the particles DB. >>>>>>>>>>> Bret might had something similar implemented - but it is >>>>>>>>>>> probably >>>>>>>>>>> obsolete. >>>>>>>>>>> IMP.Atom will probably need such functionality as well. >>>>>>>>>>> has anyone took a look at that before ? >>>>>>>>>>> thank you, >>>>>>>>>>> Keren. >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> IMP-dev mailing list >>>>>>>>>>> IMP-dev@salilab.org >>>>>>>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> IMP-dev mailing list >>>>>>>>>> IMP-dev@salilab.org >>>>>>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> IMP-dev mailing list >>>>>>>>> IMP-dev@salilab.org >>>>>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> IMP-dev mailing list >>>>>>>> IMP-dev@salilab.org >>>>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>>>> >>>>>>> _______________________________________________ >>>>>>> IMP-dev mailing list >>>>>>> IMP-dev@salilab.org >>>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>>> >>>>>> _______________________________________________ >>>>>> IMP-dev mailing list >>>>>> IMP-dev@salilab.org >>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>> >>>>> _______________________________________________ >>>>> IMP-dev mailing list >>>>> IMP-dev@salilab.org >>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>> >>>> _______________________________________________ >>>> IMP-dev mailing list >>>> IMP-dev@salilab.org >>>> https://salilab.org/mailman/listinfo/imp-dev >>> >>> _______________________________________________ >>> IMP-dev mailing list >>> IMP-dev@salilab.org >>> https://salilab.org/mailman/listinfo/imp-dev >>> >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev@salilab.org >> https://salilab.org/mailman/listinfo/imp-dev > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev

looks reasonable. Q1: can you indicate the relationship between the attributes ( and/or) ? Q2: can it support range ( all residues in a range for example) Q3: how do you support hierarchy? should the container hold all the leaves ?

On Tue, 9 Dec 2008, Daniel Russel wrote:

> So does something like this look to be along the lines of what you all have > in mind? > //! Create a map from attributes to particles > /** The map can be built with up to 4 different attributes of type Int or > String. Particles missing the named attributes are skipped. > */ > template <class TypeA, class TypeB=NullType, class TypeC=NullType, class > TypeD=NullType> > class MapScoreState: public RefCountedObject { > public: > typedef fancy_crap Key; > typedef more_fancy_crap Value; > //! Initialize the table to index on the attributes in at > /** \param[in] pcs The container of particles to index in the table > \param[in] at The set of attributes to index on*/ > MapScoreState(ParticleContainer *pcs, const Attributes &at); > //! Get the particles matching v > Particles get_particles(const Values &v) const; > //! Get the particle matching v > /** \throw InvalidStateExpeption if more than one particle matches > */ > Particle *get_particle(const Values &v) const; > > //! Also would have the score state update method to force an update > }; > > > typedef MapScoreState<Int> AtomIndexMap; > > AtomIndexMap *myindexmap= new AtomIndexMap(myparticles, > AtomIndexMap::Key(AtomDecorator::get_index_key())); > myindexmap->before_evaluate(-1); // we might want to add an update method > to ScoreState > > Particle *atom10 = myindexmap->get_particle(AtomIndexMap::Value(10)); > > For python, we would have to create a bunch of the maps with pre-chosen > types (i.e. IntMapScoreState, IntStringMapScoreState). > > > On Dec 9, 2008, at 11:08 AM, Dina Schneidman wrote: > >> but you don't always want to update you indexing before/after scoring >> >> On Tue, Dec 9, 2008 at 11:06 AM, Daniel Russel drussel@gmail.com wrote: >>> Score states don't have anything to do with scoring either :-) they are >>> just >>> updated before scoring since that is when things can change during >>> optimization. They used to just be called States which is perhaps >>> clearer. >>> >>> >>> >>> On Dec 9, 2008, at 10:56 AM, "Dina Schneidman" duhovka@gmail.com >>> wrote: >>> >>>> maybe it's a simple solution in order to have it in a model, but >>>> conceptually this indexing has nothing to do with scoring >>>> >>>> On Tue, Dec 9, 2008 at 10:42 AM, Daniel Russel drussel@gmail.com >>>> wrote: >>>>> >>>>> From what I understand, what you want is a way of specifying what >>>>> indexes >>>>> we >>>>> want build (not a way of specifying queries). We could easily provide >>>>> ScoreStates for indexes based on: >>>>> - set of discrete valued attributes >>>>> - D-dimensional interval queries on float values >>>>> >>>>> >>>>> On Dec 9, 2008, at 9:55 AM, Keren Lasker wrote: >>>>> >>>>>> Daniel - we all know how to run for loops ;) >>>>>> I just thought it make sense to have something more efficient :) >>>>>> >>>>>> On Tue, 9 Dec 2008, Daniel Russel wrote: >>>>>> >>>>>>> >>>>>>> On Dec 9, 2008, at 9:29 AM, Keren Lasker wrote: >>>>>>> >>>>>>>> For the 26S project we currently do: >>>>>>>> get particle by name >>>>>>> >>>>>>> How could we beat something like: >>>>>>> [x for x in myparticles if x.has_attribute(name) and >>>>>>> x.get_value(name) >>>>>>> == >>>>>>> "myname"] >>>>>>> with an SQL query? >>>>>>>> >>>>>>>> get a set of particles within a residue number range >>>>>>> >>>>>>> again, some variant on: >>>>>>> [x for x in molecular_hierarchy_get_by_type(root, >>>>>>> MolecularHierarchyDecorator.RESIDUE) if >>>>>>> ResidueDecorator(x).get_index() > lb and >>>>>>> ResidueDecorator(x).get_index() >>>>>>> <ub] >>>>>>> >>>>>>> or C++, something like >>>>>>> BOOST_FOREACH(Particle *p, molecular_hierarchy_get_by_type(root, >>>>>>> MolecularHierarchyDecorator.RESIDUE)) { >>>>>>> if (ResidueDecorator(p).get_index() > lb and >>>>>>> ResidueDecorator(p).get_index() <ub) { >>>>>>> // do something >>>>>>> } >>>>>>> } >>>>>>>> >>>>>>>> the solution of a function that gets (key,value,container) seems >>>>>>>> like >>>>>>>> a >>>>>>>> good solution. >>>>>>>> However - it can be more complicated : >>>>>>>> 1. it can interact with the hierarchy - give me the residues range >>>>>>>> within this protein for example - so we should probably also allow >>>>>>>> for >>>>>>>> a >>>>>>>> hierarchy handle in the interface. >>>>>>>> 2. we might want to ask residue range + some other property such as >>>>>>>> have >>>>>>>> structural coverage or do not. Therefore I think that a sql type >>>>>>>> string can >>>>>>>> be more general than a list of attributes - because you do not know >>>>>>>> how they >>>>>>>> are related. >>>>>>> >>>>>>> But the added complication is why I would suggest sticking with C++ >>>>>>> or >>>>>>> python. Lambda functions or list comprehensions support very general >>>>>>> logic >>>>>>> (more so than SQL) and allow you to leverage existing code. SQL >>>>>>> would >>>>>>> make >>>>>>> it really hard to use any of the existing functionality and require >>>>>>> lots of >>>>>>> things be exposed in another language. For example, try to find all >>>>>>> particles close to a point in SQL? It is kind of ugly. >>>>>>> >>>>>>>> On Tue, 9 Dec 2008, Daniel Russel wrote: >>>>>>>>> >>>>>>>>> In general, I think having queries on the whole collection of >>>>>>>>> particles >>>>>>>>> in the model is not a good idea (since other people's code, >>>>>>>>> restraints or >>>>>>>>> states can add particles to the model and you can never be sure >>>>>>>>> what >>>>>>>>> those >>>>>>>>> look like). >>>>>>>>> There is already functionality to search a Hierarchy (although it >>>>>>>>> is >>>>>>>>> more aimed at C++-- we could use a python interface which takes >>>>>>>>> takes >>>>>>>>> a >>>>>>>>> python lambda function to make it more convenient to use in >>>>>>>>> python). >>>>>>>>> And >>>>>>>>> python has all sorts of features for searching a list (and C++ has >>>>>>>>> a >>>>>>>>> few). >>>>>>>>> It is not clear to me that we could provide an interface that is >>>>>>>>> general and much more concise. >>>>>>>>> As a slight simplification for python users, we could provide a >>>>>>>>> function which takes a list of key, value pairs (with keys of >>>>>>>>> arbitrary >>>>>>>>> type) and a container. It is a bit messier to provide this >>>>>>>>> interface >>>>>>>>> in C++ >>>>>>>>> as we would have to have a separate list per type. >>>>>>>>> Another thing to simplify such search would be a >>>>>>>>> "DefaultValuesDecorator" which wraps a particle and pretend it has >>>>>>>>> all >>>>>>>>> attributes, just providing default values for missing ones. This >>>>>>>>> would >>>>>>>>> obviate the need to check for an attribute before matching against >>>>>>>>> it. >>>>>>>>> What sort of queries do you all do? >>>>>>>>> On Dec 9, 2008, at 8:43 AM, Dina Schneidman wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> I need this too (surprisingly). Usually I do it with mapping >>>>>>>>>> between >>>>>>>>>> the particle and the attribute. >>>>>>>>>> It is simple. however it is unclear where should we put such a >>>>>>>>>> mapping. Putting it in a model >>>>>>>>>> could be the best, however not everyone needs it. So it means >>>>>>>>>> somewhere else or extending the Model to ProteinModel? >>>>>>>>>> Dina >>>>>>>>>> P.S. skype me, we can talk about it >>>>>>>>>> On Tue, Dec 9, 2008 at 7:03 AM, Keren Lasker kerenl@salilab.org >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> hi all, >>>>>>>>>>> Frido and I find ourselves many times need to query particles >>>>>>>>>>> based >>>>>>>>>>> on >>>>>>>>>>> attribute values. >>>>>>>>>>> Few such examples: a protein with a specific name, particles >>>>>>>>>>> with a >>>>>>>>>>> specific >>>>>>>>>>> residue range. >>>>>>>>>>> I think that it would be very useful to have something similar >>>>>>>>>>> to >>>>>>>>>>> SQL >>>>>>>>>>> queries on the particles DB. >>>>>>>>>>> Bret might had something similar implemented - but it is >>>>>>>>>>> probably >>>>>>>>>>> obsolete. >>>>>>>>>>> IMP.Atom will probably need such functionality as well. >>>>>>>>>>> has anyone took a look at that before ? >>>>>>>>>>> thank you, >>>>>>>>>>> Keren. >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> IMP-dev mailing list >>>>>>>>>>> IMP-dev@salilab.org >>>>>>>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> IMP-dev mailing list >>>>>>>>>> IMP-dev@salilab.org >>>>>>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> IMP-dev mailing list >>>>>>>>> IMP-dev@salilab.org >>>>>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> IMP-dev mailing list >>>>>>>> IMP-dev@salilab.org >>>>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>>>> >>>>>>> _______________________________________________ >>>>>>> IMP-dev mailing list >>>>>>> IMP-dev@salilab.org >>>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>>> >>>>>> _______________________________________________ >>>>>> IMP-dev mailing list >>>>>> IMP-dev@salilab.org >>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>> >>>>> _______________________________________________ >>>>> IMP-dev mailing list >>>>> IMP-dev@salilab.org >>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>> >>>> _______________________________________________ >>>> IMP-dev mailing list >>>> IMP-dev@salilab.org >>>> https://salilab.org/mailman/listinfo/imp-dev >>> >>> _______________________________________________ >>> IMP-dev mailing list >>> IMP-dev@salilab.org >>> https://salilab.org/mailman/listinfo/imp-dev >>> >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev@salilab.org >> https://salilab.org/mailman/listinfo/imp-dev > > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev

I think such functionality would be a reasonable candidate for a new module. Especially as core is getting very large and cluttered.

Perhaps IMP.indexes (or .indices if people prefer that spelling).

On Dec 9, 2008, at 4:58 PM, Keren Lasker wrote:

> looks reasonable. > Q1: can you indicate the relationship between the attributes ( and/ > or) ? > Q2: can it support range ( all residues in a range for example) > Q3: how do you support hierarchy? should the container hold all the > leaves ? > > On Tue, 9 Dec 2008, Daniel Russel wrote: > >> So does something like this look to be along the lines of what you >> all have in mind? >> //! Create a map from attributes to particles >> /** The map can be built with up to 4 different attributes of type >> Int or >> String. Particles missing the named attributes are skipped. >> */ >> template <class TypeA, class TypeB=NullType, class TypeC=NullType, >> class TypeD=NullType> >> class MapScoreState: public RefCountedObject { >> public: >> typedef fancy_crap Key; >> typedef more_fancy_crap Value; >> //! Initialize the table to index on the attributes in at >> /** \param[in] pcs The container of particles to index in the table >> \param[in] at The set of attributes to index on*/ >> MapScoreState(ParticleContainer *pcs, const Attributes &at); >> //! Get the particles matching v >> Particles get_particles(const Values &v) const; >> //! Get the particle matching v >> /** \throw InvalidStateExpeption if more than one particle matches >> */ >> Particle *get_particle(const Values &v) const; >> >> //! Also would have the score state update method to force an update >> }; >> >> >> typedef MapScoreState<Int> AtomIndexMap; >> >> AtomIndexMap *myindexmap= new AtomIndexMap(myparticles, >> >> AtomIndexMap::Key(AtomDecorator::get_index_key())); >> myindexmap->before_evaluate(-1); // we might want to add an update >> method to ScoreState >> >> Particle *atom10 = myindexmap->get_particle(AtomIndexMap::Value(10)); >> >> For python, we would have to create a bunch of the maps with pre- >> chosen types (i.e. IntMapScoreState, IntStringMapScoreState). >> >> >> On Dec 9, 2008, at 11:08 AM, Dina Schneidman wrote: >> >>> but you don't always want to update you indexing before/after >>> scoring >>> On Tue, Dec 9, 2008 at 11:06 AM, Daniel Russel drussel@gmail.com >>> wrote: >>>> Score states don't have anything to do with scoring either :-) >>>> they are just >>>> updated before scoring since that is when things can change during >>>> optimization. They used to just be called States which is perhaps >>>> clearer. >>>> On Dec 9, 2008, at 10:56 AM, "Dina Schneidman" >>>> duhovka@gmail.com wrote: >>>>> maybe it's a simple solution in order to have it in a model, but >>>>> conceptually this indexing has nothing to do with scoring >>>>> On Tue, Dec 9, 2008 at 10:42 AM, Daniel Russel >>>>> drussel@gmail.com wrote: >>>>>> From what I understand, what you want is a way of specifying >>>>>> what indexes >>>>>> we >>>>>> want build (not a way of specifying queries). We could easily >>>>>> provide >>>>>> ScoreStates for indexes based on: >>>>>> - set of discrete valued attributes >>>>>> - D-dimensional interval queries on float values >>>>>> On Dec 9, 2008, at 9:55 AM, Keren Lasker wrote: >>>>>>> Daniel - we all know how to run for loops ;) >>>>>>> I just thought it make sense to have something more efficient :) >>>>>>> On Tue, 9 Dec 2008, Daniel Russel wrote: >>>>>>>> On Dec 9, 2008, at 9:29 AM, Keren Lasker wrote: >>>>>>>>> For the 26S project we currently do: >>>>>>>>> get particle by name >>>>>>>> How could we beat something like: >>>>>>>> [x for x in myparticles if x.has_attribute(name) and >>>>>>>> x.get_value(name) >>>>>>>> == >>>>>>>> "myname"] >>>>>>>> with an SQL query? >>>>>>>>> get a set of particles within a residue number range >>>>>>>> again, some variant on: >>>>>>>> [x for x in molecular_hierarchy_get_by_type(root, >>>>>>>> MolecularHierarchyDecorator.RESIDUE) if >>>>>>>> ResidueDecorator(x).get_index() > lb and >>>>>>>> ResidueDecorator(x).get_index() >>>>>>>> <ub] >>>>>>>> or C++, something like >>>>>>>> BOOST_FOREACH(Particle *p, >>>>>>>> molecular_hierarchy_get_by_type(root, >>>>>>>> MolecularHierarchyDecorator.RESIDUE)) { >>>>>>>> if (ResidueDecorator(p).get_index() > lb and >>>>>>>> ResidueDecorator(p).get_index() <ub) { >>>>>>>> // do something >>>>>>>> } >>>>>>>> } >>>>>>>>> the solution of a function that gets (key,value,container) >>>>>>>>> seems like >>>>>>>>> a >>>>>>>>> good solution. >>>>>>>>> However - it can be more complicated : >>>>>>>>> 1. it can interact with the hierarchy - give me the >>>>>>>>> residues range >>>>>>>>> within this protein for example - so we should probably also >>>>>>>>> allow for >>>>>>>>> a >>>>>>>>> hierarchy handle in the interface. >>>>>>>>> 2. we might want to ask residue range + some other property >>>>>>>>> such as >>>>>>>>> have >>>>>>>>> structural coverage or do not. Therefore I think that a sql >>>>>>>>> type >>>>>>>>> string can >>>>>>>>> be more general than a list of attributes - because you do >>>>>>>>> not know >>>>>>>>> how they >>>>>>>>> are related. >>>>>>>> But the added complication is why I would suggest sticking >>>>>>>> with C++ or >>>>>>>> python. Lambda functions or list comprehensions support very >>>>>>>> general >>>>>>>> logic >>>>>>>> (more so than SQL) and allow you to leverage existing code. >>>>>>>> SQL would >>>>>>>> make >>>>>>>> it really hard to use any of the existing functionality and >>>>>>>> require >>>>>>>> lots of >>>>>>>> things be exposed in another language. For example, try to >>>>>>>> find all >>>>>>>> particles close to a point in SQL? It is kind of ugly. >>>>>>>>> On Tue, 9 Dec 2008, Daniel Russel wrote: >>>>>>>>>> In general, I think having queries on the whole collection of >>>>>>>>>> particles >>>>>>>>>> in the model is not a good idea (since other people's code, >>>>>>>>>> restraints or >>>>>>>>>> states can add particles to the model and you can never be >>>>>>>>>> sure what >>>>>>>>>> those >>>>>>>>>> look like). >>>>>>>>>> There is already functionality to search a Hierarchy >>>>>>>>>> (although it is >>>>>>>>>> more aimed at C++-- we could use a python interface which >>>>>>>>>> takes takes >>>>>>>>>> a >>>>>>>>>> python lambda function to make it more convenient to use in >>>>>>>>>> python). >>>>>>>>>> And >>>>>>>>>> python has all sorts of features for searching a list (and C >>>>>>>>>> ++ has a >>>>>>>>>> few). >>>>>>>>>> It is not clear to me that we could provide an interface >>>>>>>>>> that is >>>>>>>>>> general and much more concise. >>>>>>>>>> As a slight simplification for python users, we could >>>>>>>>>> provide a >>>>>>>>>> function which takes a list of key, value pairs (with keys of >>>>>>>>>> arbitrary >>>>>>>>>> type) and a container. It is a bit messier to provide this >>>>>>>>>> interface >>>>>>>>>> in C++ >>>>>>>>>> as we would have to have a separate list per type. >>>>>>>>>> Another thing to simplify such search would be a >>>>>>>>>> "DefaultValuesDecorator" which wraps a particle and pretend >>>>>>>>>> it has >>>>>>>>>> all >>>>>>>>>> attributes, just providing default values for missing ones. >>>>>>>>>> This >>>>>>>>>> would >>>>>>>>>> obviate the need to check for an attribute before matching >>>>>>>>>> against >>>>>>>>>> it. >>>>>>>>>> What sort of queries do you all do? >>>>>>>>>> On Dec 9, 2008, at 8:43 AM, Dina Schneidman wrote: >>>>>>>>>>> Hi, >>>>>>>>>>> I need this too (surprisingly). Usually I do it with >>>>>>>>>>> mapping between >>>>>>>>>>> the particle and the attribute. >>>>>>>>>>> It is simple. however it is unclear where should we put >>>>>>>>>>> such a >>>>>>>>>>> mapping. Putting it in a model >>>>>>>>>>> could be the best, however not everyone needs it. So it >>>>>>>>>>> means >>>>>>>>>>> somewhere else or extending the Model to ProteinModel? >>>>>>>>>>> Dina >>>>>>>>>>> P.S. skype me, we can talk about it >>>>>>>>>>> On Tue, Dec 9, 2008 at 7:03 AM, Keren Lasker <kerenl@salilab.org >>>>>>>>>>> > >>>>>>>>>>> wrote: >>>>>>>>>>>> hi all, >>>>>>>>>>>> Frido and I find ourselves many times need to query >>>>>>>>>>>> particles based >>>>>>>>>>>> on >>>>>>>>>>>> attribute values. >>>>>>>>>>>> Few such examples: a protein with a specific name, >>>>>>>>>>>> particles with a >>>>>>>>>>>> specific >>>>>>>>>>>> residue range. >>>>>>>>>>>> I think that it would be very useful to have something >>>>>>>>>>>> similar to >>>>>>>>>>>> SQL >>>>>>>>>>>> queries on the particles DB. >>>>>>>>>>>> Bret might had something similar implemented - but it is >>>>>>>>>>>> probably >>>>>>>>>>>> obsolete. >>>>>>>>>>>> IMP.Atom will probably need such functionality as well. >>>>>>>>>>>> has anyone took a look at that before ? >>>>>>>>>>>> thank you, >>>>>>>>>>>> Keren. >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> IMP-dev mailing list >>>>>>>>>>>> IMP-dev@salilab.org >>>>>>>>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> IMP-dev mailing list >>>>>>>>>>> IMP-dev@salilab.org >>>>>>>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>>>>>>> _______________________________________________ >>>>>>>>>> IMP-dev mailing list >>>>>>>>>> IMP-dev@salilab.org >>>>>>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>>>>>> _______________________________________________ >>>>>>>>> IMP-dev mailing list >>>>>>>>> IMP-dev@salilab.org >>>>>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>>>>> _______________________________________________ >>>>>>>> IMP-dev mailing list >>>>>>>> IMP-dev@salilab.org >>>>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>>>> _______________________________________________ >>>>>>> IMP-dev mailing list >>>>>>> IMP-dev@salilab.org >>>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>>> _______________________________________________ >>>>>> IMP-dev mailing list >>>>>> IMP-dev@salilab.org >>>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>> _______________________________________________ >>>>> IMP-dev mailing list >>>>> IMP-dev@salilab.org >>>>> https://salilab.org/mailman/listinfo/imp-dev >>>> _______________________________________________ >>>> IMP-dev mailing list >>>> IMP-dev@salilab.org >>>> https://salilab.org/mailman/listinfo/imp-dev >>> _______________________________________________ >>> IMP-dev mailing list >>> IMP-dev@salilab.org >>> https://salilab.org/mailman/listinfo/imp-dev >> >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev@salilab.org >> https://salilab.org/mailman/listinfo/imp-dev > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev

On Dec 9, 2008, at 9:53 AM, Dina Schneidman wrote:

> I think IndexModel is the best solution, but in C++. having python and > c++ is more than enough, we don't want to support something else. I > suggested extending the Model because of the update reason, in this > case you can update the indexing when you add/remove particles. I don't understand what you mean.

> > > On Tue, Dec 9, 2008 at 9:29 AM, Keren Lasker kerenl@salilab.org > wrote: >> For the 26S project we currently do: >> get particle by name >> get a set of particles within a residue number range >> >> >> the solution of a function that gets (key,value,container) seems >> like a good >> solution. >> However - it can be more complicated : >> 1. it can interact with the hierarchy - give me the residues range >> within >> this protein for example - so we should probably also allow for a >> hierarchy >> handle in the interface. >> 2. we might want to ask residue range + some other property such as >> have >> structural coverage or do not. Therefore I think that a sql type >> string can >> be more general than a list of attributes - because you do not know >> how they >> are related. >> >> >> On Tue, 9 Dec 2008, Daniel Russel wrote: >> >>> In general, I think having queries on the whole collection of >>> particles in >>> the model is not a good idea (since other people's code, >>> restraints or >>> states can add particles to the model and you can never be sure >>> what those >>> look like). >>> >>> There is already functionality to search a Hierarchy (although it >>> is more >>> aimed at C++-- we could use a python interface which takes takes a >>> python >>> lambda function to make it more convenient to use in python). And >>> python has >>> all sorts of features for searching a list (and C++ has a few). >>> It is not clear to me that we could provide an interface that is >>> general >>> and much more concise. >>> >>> As a slight simplification for python users, we could provide a >>> function >>> which takes a list of key, value pairs (with keys of arbitrary >>> type) and a >>> container. It is a bit messier to provide this interface in C++ as >>> we would >>> have to have a separate list per type. >>> >>> Another thing to simplify such search would be a >>> "DefaultValuesDecorator" >>> which wraps a particle and pretend it has all attributes, just >>> providing >>> default values for missing ones. This would obviate the need to >>> check for an >>> attribute before matching against it. >>> >>> What sort of queries do you all do? >>> >>> >>> >>> On Dec 9, 2008, at 8:43 AM, Dina Schneidman wrote: >>> >>>> Hi, >>>> >>>> I need this too (surprisingly). Usually I do it with mapping >>>> between >>>> the particle and the attribute. >>>> It is simple. however it is unclear where should we put such a >>>> mapping. Putting it in a model >>>> could be the best, however not everyone needs it. So it means >>>> somewhere else or extending the Model to ProteinModel? >>>> >>>> Dina >>>> >>>> P.S. skype me, we can talk about it >>>> >>>> >>>> On Tue, Dec 9, 2008 at 7:03 AM, Keren Lasker kerenl@salilab.org >>>> wrote: >>>>> >>>>> hi all, >>>>> >>>>> Frido and I find ourselves many times need to query particles >>>>> based on >>>>> attribute values. >>>>> Few such examples: a protein with a specific name, particles >>>>> with a >>>>> specific >>>>> residue range. >>>>> >>>>> I think that it would be very useful to have something similar >>>>> to SQL >>>>> queries on the particles DB. >>>>> Bret might had something similar implemented - but it is probably >>>>> obsolete. >>>>> IMP.Atom will probably need such functionality as well. >>>>> >>>>> has anyone took a look at that before ? >>>>> >>>>> thank you, >>>>> Keren. >>>>> _______________________________________________ >>>>> IMP-dev mailing list >>>>> IMP-dev@salilab.org >>>>> https://salilab.org/mailman/listinfo/imp-dev >>>>> >>>> _______________________________________________ >>>> IMP-dev mailing list >>>> IMP-dev@salilab.org >>>> https://salilab.org/mailman/listinfo/imp-dev >>> >>> _______________________________________________ >>> IMP-dev mailing list >>> IMP-dev@salilab.org >>> https://salilab.org/mailman/listinfo/imp-dev >> >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev@salilab.org >> https://salilab.org/mailman/listinfo/imp-dev >> > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev

we do the same - but this is an ad hoc solution. It should not be part of Model I think but an external IndexModel class maybe or a IndexDecorator. I suggested SQL because your query can be complex containing often more than one attribute. Ben - you will need it in IMP.Atom - have you already started working on that?

On Tue, 9 Dec 2008, Dina Schneidman wrote:

> Hi, > > I need this too (surprisingly). Usually I do it with mapping between > the particle and the attribute. > It is simple. however it is unclear where should we put such a > mapping. Putting it in a model > could be the best, however not everyone needs it. So it means > somewhere else or extending the Model to ProteinModel? > > Dina > > P.S. skype me, we can talk about it > > > On Tue, Dec 9, 2008 at 7:03 AM, Keren Lasker kerenl@salilab.org wrote: >> hi all, >> >> Frido and I find ourselves many times need to query particles based on >> attribute values. >> Few such examples: a protein with a specific name, particles with a specific >> residue range. >> >> I think that it would be very useful to have something similar to SQL >> queries on the particles DB. >> Bret might had something similar implemented - but it is probably obsolete. >> IMP.Atom will probably need such functionality as well. >> >> has anyone took a look at that before ? >> >> thank you, >> Keren. >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev@salilab.org >> https://salilab.org/mailman/listinfo/imp-dev >> > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev >

replied to this question in the email I have just sent :)

On Tue, 9 Dec 2008, Daniel Russel wrote:

> Why use SQL over python or C++ logic? > > On Dec 9, 2008, at 9:15 AM, Keren Lasker wrote: > >> we do the same - but this is an ad hoc solution. > Agreed, but it might be one of those things (like storing the whole model, > restraints and all) where there is unlikely to be a general solution that > adds any benefit as the general case is very complex and the ad hoc solution > is quite simple. > >> >> It should not be part of Model I think but an external IndexModel class >> maybe or a IndexDecorator. > Again, I think having it act on a whole model is a really bad idea (the IMP > equivalent of using global variables :-) > >> >> I suggested SQL because your query can be complex containing often more >> than one attribute. > Why not just use python or C++ rather than add another language to learn? > There are a few operations which are more concise in SQL, but I don't know > that we want to support them or that they make sense in our case (where would > presumably have to return whole particles). > >> >> Ben - you will need it in IMP.Atom - have you already started working on >> that? > How come? I think I am missing some use case :-) > >> >> >> On Tue, 9 Dec 2008, Dina Schneidman wrote: >> >>> Hi, >>> >>> I need this too (surprisingly). Usually I do it with mapping between >>> the particle and the attribute. >>> It is simple. however it is unclear where should we put such a >>> mapping. Putting it in a model >>> could be the best, however not everyone needs it. So it means >>> somewhere else or extending the Model to ProteinModel? >>> >>> Dina >>> >>> P.S. skype me, we can talk about it >>> >>> >>> On Tue, Dec 9, 2008 at 7:03 AM, Keren Lasker kerenl@salilab.org wrote: >>>> hi all, >>>> >>>> Frido and I find ourselves many times need to query particles based on >>>> attribute values. >>>> Few such examples: a protein with a specific name, particles with a >>>> specific >>>> residue range. >>>> >>>> I think that it would be very useful to have something similar to SQL >>>> queries on the particles DB. >>>> Bret might had something similar implemented - but it is probably >>>> obsolete. >>>> IMP.Atom will probably need such functionality as well. >>>> >>>> has anyone took a look at that before ? >>>> >>>> thank you, >>>> Keren. >>>> _______________________________________________ >>>> IMP-dev mailing list >>>> IMP-dev@salilab.org >>>> https://salilab.org/mailman/listinfo/imp-dev >>>> >>> _______________________________________________ >>> IMP-dev mailing list >>> IMP-dev@salilab.org >>> https://salilab.org/mailman/listinfo/imp-dev >>> >> _______________________________________________ >> IMP-dev mailing list >> IMP-dev@salilab.org >> https://salilab.org/mailman/listinfo/imp-dev > > _______________________________________________ > IMP-dev mailing list > IMP-dev@salilab.org > https://salilab.org/mailman/listinfo/imp-dev