Most importantly, Modeller only has information in its database for
alignments of standard amino acids. So if you build a model with a
non-standard residue, it will basically ignore the alignment - it'll
handle it very similarly to a ligand. For a modified amino acid, it
probably makes most sense to apply the modification to the final model.
If for some reason you really don't want to do that, you'd be better off
making and applying a patch rather than creating a whole new residue.
If this is the case, how can modifying both the top.lib and restyp.lib
to add the nonstandard amino acids information make any sense?
Generally the only time this makes sense is if you want to model a
ligand flexibly (and even then unless you are very careful with your
parameterization, it's not going to work as well as the standard
behavior of keeping the ligand rigid).