I am trying to force a helical segment (that was built from the
non-aligned part of the sequence) to be roughly between two other
helical segments rather than sticking outside as automodel/loopmodel
puts it by default. I have tried to create an angle restraint between
the segment centroids but it seems to be ignored (apart from reporting
its violation in the log).
If it's in the log, it's in the scoring function, so it's not being
ignored. And your code looks fine to me. But you probably still have
conflicting restraints, derived from the alignment. The usual way to
resolve that is to tweak the alignment some more (one option would be to
use multiple templates, one for each domain, to allow them to move
relative to each other). You already said your helix is in a non-aligned
region, but likely it's being affected by other aligned parts of the
sequence.