Groups hangs in index backup

[Iago77]

Hi all,

one month ago we performed an upgrade from Networker 7.3 SP3 to 7.4 SP5.

Scheduled backups don't complete, although the majority of the saveset does, but the index backups are hung indefinitely. This affects the rest of the subsequent executions on the same group, for example full scheduling in the weekend and incremmental from Monday to Friday.

I have changed the jobsdb retention parameters: 7 days and 200 MB.

Besides, until now I wasn't able to stop the group from GUI, but since I increased the number of file descriptors per user, nproc, from 1024 to 8192 this problem hasn't appeared again.

There are about 10 groups splitted in several star-times, with many saveset each one.

Many thanks in advance for you help.

Iago

 

[a060463xyz]

run savegrp -vO name_of_group
and post the output here.

 

[Iago77]

It's a pretty long log file, so here you are.

It seems it finishes correctly, but in the night it often fails:

32451:savegrp: datos1:/descargas level=incr
32451:savegrp: datos1:/ora10h level=incr
32451:savegrp: datos1:/ora92h level=incr
32451:savegrp: datos1:/soft/components/oracle level=incr
32451:savegrp: datos1:/soft/components/oracle10 level=incr
32451:savegrp: datos1:/soft/oracle level=incr
32451:savegrp: datos1:/soft/oracle/agent10g level=incr
32451:savegrp: datos1:/soft/oracle/ora10 level=incr
32451:savegrp: datos1:/soft/oracle/ora102 level=incr
32451:savegrp: datos1:/soft/oracle/ora92 level=incr
32451:savegrp: datos2:/cargas level=incr
32451:savegrp: datos2:/ora10h level=incr
32451:savegrp: datos2:/ora92h level=incr
32451:savegrp: datos2:/soft/oracle level=incr
32451:savegrp: datos2:/soft/oracle/agent10g level=incr
32451:savegrp: datos2:/soft/oracle/ora10 level=incr
32451:savegrp: poseidon:/ora817 level=incr
32451:savegrp: poseidon:/mistral level=incr
32451:savegrp: poseidon:/export/home/misobj level=incr
32451:savegrp: poseidon:/imagenes level=incr
32451:savegrp: poseidon:/ism level=incr
32451:savegrp: poseidon:/iws/ompi level=incr
32451:savegrp: poseidon:/var/spool/mistralx level=incr
32451:savegrp: poseidon:/imapodgen level=incr
32451:savegrp: poseidon:/java level=incr
32451:savegrp: poseidon:/mtrabajo level=incr
32451:savegrp: poseidon:/produccion/ora8 level=incr
32451:savegrp: poseidon:/pruebas/ora8 level=incr
32451:savegrp: poseidon:/desarrollo level=incr
32451:savegrp: urano:/bck_bopi level=incr
32451:savegrp: urano:/home/difsel level=incr
32451:savegrp: urano:/home/general level=incr
32451:savegrp: urano:/home/general/logs level=incr
32451:savegrp: urano:/home/nmarcas level=incr
32451:savegrp: urano:/oraclecs level=incr
32451:savegrp: urano:/soft level=incr
32451:savegrp: dlapphp05v:/home/jbadm level=incr
32451:savegrp: dlapphp05v:/var/lib/jboss level=incr
32451:savegrp: dlapphp05v:/var/log/jboss level=incr
32451:savegrp: dlapphp05v:/opt/redhat/jboss-4.2.2.GA level=incr
32451:savegrp: dlapphp05v:/srv/jboss level=incr
32451:savegrp: dlapphp05v:/opt/oracle/agent level=incr
32451:savegrp: dlapphp05v:/var/log/oracle/agent10g level=incr
32451:savegrp: virtual2:/soft/oracle level=incr
32451:savegrp: virtual2:/soft/oracle/oemg level=incr
32451:savegrp: virtual2:/software level=incr
32451:savegrp: triton:/bea level=incr
32451:savegrp: triton:/bea10 level=incr
32451:savegrp: triton:/CARGA level=incr
32451:savegrp: triton:/home/fatwire level=incr
32451:savegrp: triton:/home/LC level=incr
32451:savegrp: triton:/home/mani level=incr
32451:savegrp: triton:/home/modibu2 level=incr
32451:savegrp: triton:/home/msan level=incr
32451:savegrp: triton:/home/nmarcas2 level=incr
32451:savegrp: triton:/home/tonlinebak level=incr
32451:savegrp: triton:/home/tuxe64 level=incr
32451:savegrp: triton:/maniobra level=incr
32451:savegrp: triton:/oracle level=incr
32451:savegrp: triton:/soft level=incr
32451:savegrp: triton:/softora/ora10 level=incr
32451:savegrp: triton:/softora/ora92 level=incr
32451:savegrp: triton:/users level=incr
32451:savegrp: triton:/usr/iecisa level=incr
32451:savegrp: triton:/usr/ns-home level=incr
32451:savegrp: plapphp02f:/opt/oracle/bea10 level=incr
32451:savegrp: plapphp02f:/opt/sap/boxi31 level=incr
32451:savegrp: plapphp02f:/opt/oracle/agent level=incr
32451:savegrp: plapphp02f:/var/log/oracle/agent10g level=incr
32451:savegrp: plapphp02f:/var/log/weblogic/ALFA level=incr
32451:savegrp: plapphp02f:/var/log/weblogic/BO level=incr
32451:savegrp: plapphp02f:/var/log/weblogic/INVENES level=incr
32451:savegrp: plapphp02f:/var/log/weblogic/tramites level=incr
32451:savegrp: plapphp02f:/var/log/weblogic/Disenos level=incr
32451:savegrp: plapphp02f:/var/log/weblogic/consultas level=incr
32451:savegrp: plapphp02f:/var/log/weblogic/dticomun level=incr
32451:savegrp: plapphp02f:/var/log/weblogic/interno level=incr
32451:savegrp: plapphp02f:/var/log/weblogic/recursos level=incr
32451:savegrp: plapphp02f:/opt/oracle/bea10/user_projects/domains/WLIProDIS/servers level=incr
32451:savegrp: plapphp02f:/opt/oracle/bea10/user_projects/domains/WLIProREC/servers level=incr
32451:savegrp: plapphp02f:/opt/oracle/bea10/user_projects/domains/WLSProduccion/servers level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/Alfa level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/INVENES level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/backup level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/badaral level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/bopi level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/d_srv_wl_alfa level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/desa_dwpmdes level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/desa_var_lib_oracle_dwh level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/desarrolloWLS level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/general level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pre_ora10_scripts level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pre_ora11_scripts level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pre_srv_ALFA level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pre_srv_BO level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pre_srv_INVENES level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pre_srv_consultas level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pre_srv_disenos level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pre_srv_dticomun level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pre_srv_interno level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pre_srv_tramites level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pre_var_lib_ALFA level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pre_var_lib_INVENES level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pre_var_lib_consultas level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pre_var_lib_disenos level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pre_var_lib_tramites level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_ora10_scripts level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_ora11_scripts level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_srv_ALFA level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_srv_BO level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_srv_INVENES level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_srv_consultas level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_srv_disenos level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_srv_dticomun level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_srv_interno level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_srv_recursos level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_srv_tramites level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_srv_utiles level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_var_lib_ALFA level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_var_lib_INVENES level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_var_lib_consultas level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_var_lib_disenos level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_var_lib_recursos level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_var_lib_tramites level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_wli_domains_config level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_wli_domains_dis_data level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_wli_domains_rec_config level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_wli_domains_rec_data level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_wls_domains_config level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/pro_wls_domains_data level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/produccionWLS level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/proyectosh level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/telematica level=incr
32451:savegrp: ssdmz008:/BACKUPS/nas/ucm level=incr
32451:savegrp: odilon:/servimagen level=incr
32451:savegrp: odilon:/CARGA/DIARIA level=incr
32451:savegrp: odilon:/CARGA/MASIVA level=incr
32451:savegrp: saturno:/bea level=incr
32451:savegrp: saturno:/home/general level=incr
32451:savegrp: saturno:/home/general/logs level=incr
32451:savegrp: saturno:/home/mani level=incr
32451:savegrp: saturno:/home/msan level=incr
32451:savegrp: saturno:/home/nmarcas level=incr
32451:savegrp: saturno:/home/tonline level=incr
32451:savegrp: saturno:/soft level=incr
32451:savegrp: saturno:/softora/ora10 level=incr
32451:savegrp: saturno:/softora/ora92 level=incr
32451:savegrp: plsrvhp08v:/var/

http://www/REPO

level=incr
32451:savegrp: dlapphp02f:/opt/oracle/bea10 level=incr
32451:savegrp: dlapphp02f:/srv/weblogic/ALFA level=incr
32451:savegrp: dlapphp02f:/var/log/weblogic/ALFA level=incr
32451:savegrp: dlapphp02f:/srv/weblogic/Disenos level=incr
32451:savegrp: dlapphp02f:/var/log/weblogic/Disenos level=incr
32451:savegrp: dlapphp02f:/srv/weblogic/recursos level=incr
32451:savegrp: dlapphp02f:/var/log/weblogic/recursos level=incr
32451:savegrp: dlapphp02f:/srv/weblogic/tribunales level=incr
32451:savegrp: dlapphp02f:/var/log/weblogic/tribunales level=incr
32451:savegrp: dlapphp02f:/opt/oracle/bea10/user_projects/domains/WLIDesDIS/servers level=incr
32451:savegrp: dlapphp02f:/opt/oracle/bea10/user_projects/domains/WLIDesALF/servers level=incr
32451:savegrp: dlapphp02f:/opt/oracle/bea10/user_projects/domains/WLIDesREC/servers level=incr
32451:savegrp: dlapphp02f:/opt/oracle/agent level=incr
32451:savegrp: dlapphp02f:/var/log/oracle/agent10g level=incr
32451:savegrp: plstohp01f.oepm.local:NMCASA:/gst_on_plstohp01f/lgto_gst level=full
32451:savegrp: dlapphp07v:/usr/local/tmview level=incr
32451:savegrp: dlapphp07v:/var/log/tmview level=incr
32451:savegrp: dlapphp07v:/var/lib/tmview level=incr
32451:savegrp: plappfj03f:/usr/local/tmview level=incr
32451:savegrp: plappfj03f:/var/log/tmview level=incr
32451:savegrp: plappfj03f:/var/lib/tmview level=incr
32451:savegrp: plapphp06v:/opt/redhat/jboss-4.2.2.GA level=incr
32451:savegrp: plapphp06v:/srv/jboss level=incr
32451:savegrp: plapphp06v:/var/log/jboss level=incr
32451:savegrp: plapphp06v:/var/lib/jboss level=incr
32451:savegrp: plapphp06v:/home/sigem level=incr
32451:savegrp: plapphp06v:/var/lib/ftp/sigem level=incr
32451:savegrp: plapphp06v:/opt/oracle/agent level=incr
32451:savegrp: plapphp06v:/var/log/oracle/agent10g level=incr
32451:savegrp: virtual3:/soft/oracle/agent10g level=incr
32451:savegrp: virtual3:/soft/oracle/ora92 level=incr
32451:savegrp: virtual3:/softbase/oracle level=incr
32451:savegrp: plorahp01f:/soft/oracle/product/diag level=incr
32451:savegrp: plorahp01f:/soft/oracle/product/ora10/dbs_1 level=incr
32451:savegrp: plorahp01f:/soft/oracle/product/ora11/agent_1 level=incr
32451:savegrp: plorahp01f:/soft/oracle/product/ora11/asm_1 level=incr
32451:savegrp: plorahp01f:/soft/oracle/product/ora11/crs_1 level=incr
32451:savegrp: plorahp01f:/soft/oracle/product/ora11/dbs_1 level=incr
32451:savegrp: plorahp01f:/soft/oracle/product/oraInventory level=incr
32451:savegrp: es00dbma:/work/verity_data/backup level=skip
32451:savegrp: es00dbma:/work/verity_data/colls level=skip
32451:savegrp: es00dbma:/work/verity_data/fulltext level=skip
32451:savegrp: es00dbma:/work/verity_data/images level=skip
32451:savegrp: es00dbma:/work/verity_data/scripts level=skip
32451:savegrp: es00dbma:/work/verity_data/stylesets level=skip
32451:savegrp: qlorahp02f:/home/ora11 level=incr
32451:savegrp: qlorahp02f:/soft/oracle/product/diag level=incr
32451:savegrp: qlorahp02f:/soft/oracle/product/ora10/dbs_1 level=incr
32451:savegrp: qlorahp02f:/soft/oracle/product/ora11/agent_1 level=incr
32451:savegrp: qlorahp02f:/soft/oracle/product/ora11/asm_1 level=incr
32451:savegrp: qlorahp02f:/soft/oracle/product/ora11/crs_1 level=incr
32451:savegrp: qlorahp02f:/soft/oracle/product/ora11/dbs_1 level=incr
32451:savegrp: qlorahp02f:/soft/oracle/product/oraInventory level=incr
32451:savegrp: dlapphp01v:/opt/app/bea9 level=incr
32451:savegrp: dlapphp01v:/srv/weblogic/tramites level=incr
32451:savegrp: dlapphp01v:/srv/weblogic/INVENES level=incr
32451:savegrp: dlapphp01v:/opt/oracle/bea10 level=incr
32451:savegrp: dlapphp01v:/opt/oracle/agent level=incr
32451:savegrp: dlapphp01v:/srv/weblogic/consultas level=incr
32451:savegrp: dlapphp01v:/var/log/weblogic/consultas level=incr
32451:savegrp: dlapphp01v:/var/log/weblogic/tramites level=incr
32451:savegrp: dlapphp01v:/var/log/weblogic/INVENES level=incr
32451:savegrp: dlapphp01v:/var/log/weblogic/interno level=incr
32451:savegrp: dlapphp01v:/srv/weblogic/interno level=incr
32451:savegrp: dlapphp01v:/var/log/oracle/agent10g level=incr
32451:savegrp: datos3:/ora10h level=incr
32451:savegrp: datos3:/soft/components/oracle level=incr
32451:savegrp: datos3:/soft/oracle level=incr
32451:savegrp: datos3:/soft/oracle/agent10g level=incr
32451:savegrp: datos3:/soft/oracle/ora10 level=incr
32451:savegrp: datos3:/soft/oracle/ora102 level=incr
32451:savegrp: dlapphp03v:/opt level=incr
32451:savegrp: dlapphp03v:/mnt/oracle level=incr
32451:savegrp: qlapphp01f:/opt/oracle/agent level=incr
32451:savegrp: qlapphp01f:/opt/oracle/bea10 level=incr
32451:savegrp: qlapphp01f:/opt/oracle/bea10/user_projects/domains/WLIPreALF/servers level=incr
32451:savegrp: qlapphp01f:/opt/oracle/bea10/user_projects/domains/WLIPreREC/servers level=incr
32451:savegrp: qlapphp01f:/opt/sap/boxi31 level=incr
32451:savegrp: qlapphp01f:/var/log/oracle/agent10g level=incr
32451:savegrp: qlapphp01f:/var/log/weblogic/ALFA level=incr
32451:savegrp: qlapphp01f:/var/log/weblogic/BO level=incr
32451:savegrp: qlapphp01f:/var/log/weblogic/consultas level=incr
32451:savegrp: qlapphp01f:/var/log/weblogic/Disenos level=incr
32451:savegrp: qlapphp01f:/var/log/weblogic/dticomun level=incr
32451:savegrp: qlapphp01f:/var/log/weblogic/interno level=incr
32451:savegrp: qlapphp01f:/var/log/weblogic/INVENES level=incr
32451:savegrp: qlapphp01f:/var/log/weblogic/tramites level=incr
32451:savegrp: ism1.oepm.local:/datos level=incr
32451:savegrp: ism1.oepm.local:/Xprint level=incr
32451:savegrp: plappfj01f:/opt level=incr
32451:savegrp: plapphp01f:/opt/oracle/bea10 level=incr
32451:savegrp: plapphp01f:/opt/sap/boxi31 level=incr
32451:savegrp: plapphp01f:/opt/oracle/agent level=incr
32451:savegrp: plapphp01f:/var/log/oracle/agent10g level=incr
32451:savegrp: plapphp01f:/var/log/weblogic/ALFA level=incr
32451:savegrp: plapphp01f:/var/log/weblogic/BO level=incr
32451:savegrp: plapphp01f:/var/log/weblogic/INVENES level=incr
32451:savegrp: plapphp01f:/var/log/weblogic/tramites level=incr
32451:savegrp: plapphp01f:/var/log/weblogic/Disenos level=incr
32451:savegrp: plapphp01f:/var/log/weblogic/consultas level=incr
32451:savegrp: plapphp01f:/var/log/weblogic/dticomun level=incr
32451:savegrp: plapphp01f:/var/log/weblogic/interno level=incr
32451:savegrp: plapphp01f:/var/log/weblogic/recursos level=incr
32451:savegrp: plapphp01f:/opt/oracle/bea10/user_projects/domains/WLIProDIS/servers level=incr
32451:savegrp: plapphp01f:/opt/oracle/bea10/user_projects/domains/WLIProREC/servers level=incr
32451:savegrp: plapphp01f:/opt/oracle/bea10/user_projects/domains/WLSProduccion/servers level=incr
32451:savegrp: dlorahp01f:/home/ora11 level=incr
32451:savegrp: dlorahp01f:/soft/oracle/product/diag level=incr
32451:savegrp: dlorahp01f:/soft/oracle/product/ora10/dbs_1 level=incr
32451:savegrp: dlorahp01f:/soft/oracle/product/ora11/agent_1 level=incr
32451:savegrp: dlorahp01f:/soft/oracle/product/ora11/asm_1 level=incr
32451:savegrp: dlorahp01f:/soft/oracle/product/ora11/crs_1 level=incr
32451:savegrp: dlorahp01f:/soft/oracle/product/ora11/dbs_1 level=incr
32451:savegrp: dlorahp01f:/soft/oracle/product/oraInventory level=incr
32451:savegrp: dlapphp02v:/opt/app/bea10 level=incr
32451:savegrp: dlapphp02v:/srv/weblogic/BO level=incr
32451:savegrp: dlapphp02v:/srv/weblogic/INVENES level=incr
32451:savegrp: dlapphp02v:/srv/weblogic/TPUnico level=incr
32451:savegrp: dlapphp02v:/srv/weblogic/tramites level=incr
32451:savegrp: dlapphp02v:/var/lib/weblogic/tramites level=incr
32451:savegrp: dlapphp02v:/var/log/weblogic/ALFA level=incr
32451:savegrp: dlapphp02v:/var/log/weblogic/BO level=incr
32451:savegrp: dlapphp02v:/var/log/weblogic/INVENES level=incr
32451:savegrp: dlapphp02v:/var/log/weblogic/TPUnico level=incr
32451:savegrp: dlapphp02v:/var/log/weblogic/tramites level=incr
32451:savegrp: venus:/ora10h level=incr
32451:savegrp: venus:/CARGA level=incr
32451:savegrp: venus:/CARGA/BOInstall level=incr
32451:savegrp: venus:/CARGA/MASIVA level=incr
32451:savegrp: venus:/CARGA/BOServer level=incr
32451:savegrp: venus:/soft/oracle/ora102 level=incr
32451:savegrp: venus:/soft/oracle/ora102_2 level=incr
32451:savegrp: plsrvhp07v:/var/

http://www/svn

level=incr
32451:savegrp: plorahp02f:/soft/oracle/product/diag level=incr
32451:savegrp: plorahp02f:/soft/oracle/product/ora10/dbs_1 level=incr
32451:savegrp: plorahp02f:/soft/oracle/product/ora11/agent_1 level=incr
32451:savegrp: plorahp02f:/soft/oracle/product/ora11/asm_1 level=incr
32451:savegrp: plorahp02f:/soft/oracle/product/ora11/crs_1 level=incr
32451:savegrp: plorahp02f:/soft/oracle/product/ora11/dbs_1 level=incr
32451:savegrp: plorahp02f:/soft/oracle/product/oraInventory level=incr
32451:savegrp: qlorahp01f:/home/ora11 level=incr
32451:savegrp: qlorahp01f:/soft/oracle/product/diag level=incr
32451:savegrp: qlorahp01f:/soft/oracle/product/ora10/dbs_1 level=incr
32451:savegrp: qlorahp01f:/soft/oracle/product/ora11/agent_1 level=incr
32451:savegrp: qlorahp01f:/soft/oracle/product/ora11/asm_1 level=incr
32451:savegrp: qlorahp01f:/soft/oracle/product/ora11/crs_1 level=incr
32451:savegrp: qlorahp01f:/soft/oracle/product/ora11/dbs_1 level=incr
32451:savegrp: qlorahp01f:/soft/oracle/product/oraInventory level=incr
32451:savegrp: plappfj02f:/opt level=incr
7236:savegrp: Group will not limit job parallelism
32494:savegrp: datos1:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391334 -l 9 -LL -W 78 -N index:0da61793-00000004-42526670-426e03d7-0dbf0000-ac100007 /nsr/index/datos1
32494:savegrp: datos2:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391297 -l 9 -LL -W 78 -N index:ab979050-00000004-41e3cd80-421da2cf-0f820000-ac100007 /nsr/index/datos2
32494:savegrp: poseidon:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391374 -l 9 -LL -W 78 -N index:90e2f449-00000004-3faf8546-3faf854f-00090000-ac100007 /nsr/index/poseidon
32494:savegrp: urano:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391113 -l 9 -LL -W 78 -N index:33388541-00000004-4147fc25-4147fc24-01db0000-ac100007 /nsr/index/urano
32494:savegrp: dlapphp05v:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391272 -l 9 -LL -W 78 -N index:acb71415-00000004-4acb5149-4addb96c-381a0000-ac100008 /nsr/index/dlapphp05v
32494:savegrp: virtual2:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391129 -l 9 -LL -W 78 -N index:61871640-00000004-45c846a1-45c846a0-06750000-ac100007 /nsr/index/virtual2
32494:savegrp: triton:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391103 -l 9 -LL -W 78 -N index:42c75cff-00000004-400bd404-400bd403-03b70000-ac100007 /nsr/index/triton
32494:savegrp: plapphp02f:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1260580800 -l 9 -LL -W 78 -N index:39f73236-00000004-4b041d44-4b041ecd-045e0000-ac100008 /nsr/index/plapphp02f
32494:savegrp: ssdmz008:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391299 -l 9 -LL -W 78 -N index:6ec96aaa-00000004-4714c65e-4714c65d-02080000-ac100007 /nsr/index/ssdmz008
32494:savegrp: odilon:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391138 -l 9 -LL -W 78 -N index:0ac5ace8-00000004-3faf8544-3faf8543-00070000-ac100007 /nsr/index/odilon
32494:savegrp: saturno:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391105 -l 9 -LL -W 78 -N index:bb48f449-00000004-4147fc26-4147fc50-01dc0000-ac100007 /nsr/index/saturno
32494:savegrp: plsrvhp08v:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1260586308 -l 9 -LL -W 78 -N index:803d3262-00000004-4b0e9dce-4b165028-19290000-c0a80b0f /nsr/index/plsrvhp08v
32494:savegrp: dlapphp02f:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391358 -l 9 -LL -W 78 -N index:c3a51a86-00000004-4acb5146-4ad58422-209c0000-ac100008 /nsr/index/dlapphp02f
32494:savegrp: plstohp01f.oepm.local:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -l full -LL -W 78 -N index:4765f507-00000004-4b0d08b5-4b0d08b4-00010000-c0a80b0f /nsr/index/plstohp01f.oepm.local
32494:savegrp: dlapphp07v:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1260580031 -l 9 -LL -W 78 -N index:0cdb350a-00000004-4b041d47-4b0c0009-1dc60000-ac100008 /nsr/index/dlapphp07v
32494:savegrp: plappfj03f:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1260580025 -l 9 -LL -W 78 -N index:db7b3150-00000004-4b0e9dcd-4b154adc-14560000-c0a80b0f /nsr/index/plappfj03f
32494:savegrp: plapphp06v:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391162 -l 9 -LL -W 78 -N index:bb3057ac-00000004-4acb5147-4ad5ed47-20dc0000-ac100008 /nsr/index/plapphp06v
32494:savegrp: virtual3:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391100 -l 9 -LL -W 78 -N index:083d9237-00000004-45506ca5-45506ca4-07100000-ac100007 /nsr/index/virtual3
32494:savegrp: plorahp01f:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391230 -l 9 -LL -W 78 -N index:5f4c41d1-00000004-4a35f9e3-4a35f9e2-00010000-ac100008 /nsr/index/plorahp01f
32494:savegrp: qlorahp02f:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391229 -l 9 -LL -W 78 -N index:abbe3898-00000004-4a0aa0ad-4a0aa0ac-0ce90000-ac100008 /nsr/index/qlorahp02f
32494:savegrp: dlapphp01v:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391110 -l 9 -LL -W 78 -N index:291172e8-00000004-49ae9eda-49b125e3-18ae0000-ac100008 /nsr/index/dlapphp01v
32494:savegrp: datos3:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391217 -l 9 -LL -W 78 -N index:708cd8d3-00000004-4476e7b7-4476e7b6-0bab0000-ac100007 /nsr/index/datos3
32494:savegrp: dlapphp03v:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391107 -l 9 -LL -W 78 -N index:79211daf-00000004-49d35146-49d35145-00190000-ac100008 /nsr/index/dlapphp03v
32494:savegrp: qlapphp01f:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391106 -l 9 -LL -W 78 -N index:b21b02d6-00000004-4a38a6ef-4a38a6ee-05f60000-ac100008 /nsr/index/qlapphp01f
32494:savegrp: ism1.oepm.local:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391288 -l 9 -LL -W 78 -N index:3fbedcda-00000004-3faf8541-3faf8540-00040000-ac100007 /nsr/index/ism1.oepm.local
32494:savegrp: plappfj01f:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391137 -l 9 -LL -W 78 -N index:084cd82d-00000004-49b8e996-49b8e995-09780000-ac100008 /nsr/index/plappfj01f
32494:savegrp: plapphp01f:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1260580019 -l 9 -LL -W 78 -N index:edc0c6e2-00000004-4b041d43-4b041d42-045d0000-ac100008 /nsr/index/plapphp01f
32494:savegrp: dlorahp01f:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391228 -l 9 -LL -W 78 -N index:1063ecd3-00000004-4a2d27ba-4a32363b-44d40000-ac100008 /nsr/index/dlorahp01f
32494:savegrp: dlapphp02v:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391104 -l 9 -LL -W 78 -N index:b1e0ebd5-00000004-49ae9ed9-49b125a3-18ad0000-ac100008 /nsr/index/dlapphp02v
32494:savegrp: venus:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391099 -l 9 -LL -W 78 -N index:4a799e7c-00000004-42b0534d-42b0534c-08b50000-ac100007 /nsr/index/venus
32494:savegrp: plsrvhp07v:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391166 -l 9 -LL -W 78 -N index:41cd7f97-00000004-4a4df9a1-4a66cbd1-5c450000-ac100008 /nsr/index/plsrvhp07v
32494:savegrp: plorahp02f:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391109 -l 9 -LL -W 78 -N index:b81fbb30-00000004-4a0aa0ae-4a0aa0e9-0cea0000-ac100008 /nsr/index/plorahp02f
32494:savegrp: qlorahp01f:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391146 -l 9 -LL -W 78 -N index:f69d6a2a-00000004-4a2d27b9-4a2d27b8-38270000-ac100008 /nsr/index/qlorahp01f
32494:savegrp: plappfj02f:index started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -t 1261391164 -l 9 -LL -W 78 -N index:10dc98c3-00000004-49d35147-49d35462-001a0000-ac100008 /nsr/index/plappfj02f
savegrp:UnixDatos * ssdmz008:index See the file /nsr/tmp/sg/UnixDatos/sso.ssdmz008.ZC8FEb for output of the job.
7341:savegrp: ssdmz008:index succeeded.
savegrp:UnixDatos * odilon:index See the file /nsr/tmp/sg/UnixDatos/sso.odilon.3xc9vh for output of the job.
7341:savegrp: odilon:index succeeded.
savegrp:UnixDatos * triton:index See the file /nsr/tmp/sg/UnixDatos/sso.triton.fvQg7Z for output of the job.
7341:savegrp: triton:index succeeded.
savegrp:UnixDatos * urano:index See the file /nsr/tmp/sg/UnixDatos/sso.urano.bA2ibJ for output of the job.
7341:savegrp: urano:index succeeded.
savegrp:UnixDatos * plsrvhp08v:index See the file /nsr/tmp/sg/UnixDatos/sso.plsrvhp08v.vJAyqt for output of the job.
7341:savegrp: plsrvhp08v:index succeeded.
savegrp:UnixDatos * saturno:index See the file /nsr/tmp/sg/UnixDatos/sso.saturno.xgnwrn for output of the job.
7341:savegrp: saturno:index succeeded.
savegrp:UnixDatos * dlapphp02f:index See the file /nsr/tmp/sg/UnixDatos/sso.dlapphp02f.bCqVsz for output of the job.
7341:savegrp: dlapphp02f:index succeeded.
savegrp:UnixDatos * dlapphp07v:index See the file /nsr/tmp/sg/UnixDatos/sso.dlapphp07v.fJ9cGL for output of the job.
7341:savegrp: dlapphp07v:index succeeded.
savegrp:UnixDatos * plstohp01f.oepm.local:index See the file /nsr/tmp/sg/UnixDatos/sso.plstohp01f.oepm.local.P96iyF for output of the job.
7341:savegrp: plstohp01f.oepm.local:index succeeded.
savegrp:UnixDatos * plappfj03f:index See the file /nsr/tmp/sg/UnixDatos/sso.plappfj03f.vmpxRR for output of the job.
7341:savegrp: plappfj03f:index succeeded.
savegrp:UnixDatos * plapphp06v:index See the file /nsr/tmp/sg/UnixDatos/sso.plapphp06v.DaXr5X for output of the job.
7341:savegrp: plapphp06v:index succeeded.
savegrp:UnixDatos * virtual3:index See the file /nsr/tmp/sg/UnixDatos/sso.virtual3.5jkxm4 for output of the job.
7341:savegrp: virtual3:index succeeded.
savegrp:UnixDatos * plorahp01f:index See the file /nsr/tmp/sg/UnixDatos/sso.plorahp01f.NpqvGa for output of the job.
7341:savegrp: plorahp01f:index succeeded.
savegrp:UnixDatos * qlorahp02f:index See the file /nsr/tmp/sg/UnixDatos/sso.qlorahp02f.hxDm2g for output of the job.
7341:savegrp: qlorahp02f:index succeeded.
savegrp:UnixDatos * dlapphp01v:index See the file /nsr/tmp/sg/UnixDatos/sso.dlapphp01v.7FK5pn for output of the job.
7341:savegrp: dlapphp01v:index succeeded.
savegrp:UnixDatos * dlapphp03v:index See the file /nsr/tmp/sg/UnixDatos/sso.dlapphp03v.JGRmhA for output of the job.
7341:savegrp: dlapphp03v:index succeeded.
savegrp:UnixDatos * datos3:index See the file /nsr/tmp/sg/UnixDatos/sso.datos3.bwSDPt for output of the job.
7341:savegrp: datos3:index succeeded.
savegrp:UnixDatos * qlapphp01f:index See the file /nsr/tmp/sg/UnixDatos/sso.qlapphp01f.5SOWKG for output of the job.
7341:savegrp: qlapphp01f:index succeeded.
savegrp:UnixDatos * ism1.oepm.local:index See the file /nsr/tmp/sg/UnixDatos/sso.ism1.oepm.local.LTkngN for output of the job.
7341:savegrp: ism1.oepm.local:index succeeded.
savegrp:UnixDatos * plappfj01f:index See the file /nsr/tmp/sg/UnixDatos/sso.plappfj01f.1vSyNT for output of the job.
7341:savegrp: plappfj01f:index succeeded.
savegrp:UnixDatos * dlorahp01f:index See the file /nsr/tmp/sg/UnixDatos/sso.dlorahp01f.BBcCX6 for output of the job.
7341:savegrp: dlorahp01f:index succeeded.
savegrp:UnixDatos * plapphp01f:index See the file /nsr/tmp/sg/UnixDatos/sso.plapphp01f.mwuHm0 for output of the job.
7341:savegrp: plapphp01f:index succeeded.
savegrp:UnixDatos * dlapphp02v:index See the file /nsr/tmp/sg/UnixDatos/sso.dlapphp02v.KvnwAd for output of the job.
7341:savegrp: dlapphp02v:index succeeded.
savegrp:UnixDatos * venus:index See the file /nsr/tmp/sg/UnixDatos/sso.venus.Hhccfk for output of the job.
7341:savegrp: venus:index succeeded.
savegrp:UnixDatos * qlorahp01f:index See the file /nsr/tmp/sg/UnixDatos/sso.qlorahp01f.qh0ttE for output of the job.
7341:savegrp: qlorahp01f:index succeeded.
savegrp:UnixDatos * plsrvhp07v:index See the file /nsr/tmp/sg/UnixDatos/sso.plsrvhp07v.UjzbXq for output of the job.
7341:savegrp: plsrvhp07v:index succeeded.
savegrp:UnixDatos * plappfj02f:index See the file /nsr/tmp/sg/UnixDatos/sso.plappfj02f.nHPehL for output of the job.
7341:savegrp: plappfj02f:index succeeded.
savegrp:UnixDatos * plorahp02f:index See the file /nsr/tmp/sg/UnixDatos/sso.plorahp02f.bhTlHx for output of the job.
7341:savegrp: plorahp02f:index succeeded.
savegrp:UnixDatos * dlapphp05v:index See the file /nsr/tmp/sg/UnixDatos/sso.dlapphp05v.RFkhLO for output of the job.
7341:savegrp: dlapphp05v:index succeeded.
savegrp:UnixDatos * virtual2:index See the file /nsr/tmp/sg/UnixDatos/sso.virtual2.tD5UoU for output of the job.
7341:savegrp: virtual2:index succeeded.
savegrp:UnixDatos * poseidon:index See the file /nsr/tmp/sg/UnixDatos/sso.poseidon.xafTED for output of the job.
7341:savegrp: poseidon:index succeeded.
savegrp:UnixDatos * datos1:index See the file /nsr/tmp/sg/UnixDatos/sso.datos1.JKMGIs for output of the job.
7341:savegrp: datos1:index succeeded.
savegrp:UnixDatos * datos2:index See the file /nsr/tmp/sg/UnixDatos/sso.datos2.PrJHay for output of the job.
7341:savegrp: datos2:index succeeded.
savegrp:UnixDatos * plapphp02f:index See the file /nsr/tmp/sg/UnixDatos/sso.plapphp02f.vwfeR5 for output of the job.
7341:savegrp: plapphp02f:index succeeded.
32494:savegrp: plstohp01f.oepm.local:bootstrap started
save -s plstohp01f.oepm.local -S -g UnixDatos -LL -f - -m plstohp01f.oepm.local -V -l 0 -LL -W 78 -N bootstrap -x /nsr/res /nsr/mm
savegrp:UnixDatos * plstohp01f.oepm.local:bootstrap See the file /nsr/tmp/sg/UnixDatos/sso.plstohp01f.oepm.local.dxVZfT for output of the job.
7341:savegrp: plstohp01f.oepm.local:bootstrap succeeded.
7241:savegrp: nsrim run recently, skipping

 

[605]

This looks fine so far. No prablems from what i can see.

First of all i would check whether it happens to specific or to all CFIs. Maybe it is just the case that concurrent operation will cause the problem. You may try to split into several groups.

 

[Iago77]

I have changed some kernel parameters in /etc/sysctl.conf:

net.ipv4.tcp_max_syn_backlog = 8192
net.core.netdev_max_backlog = 8192
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 8
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_fin_timeout = 60
sunrpc.tcp_slot_table_entries = 64
kernel.msgmax = 131072
kernel.shmmax = 137438953472
kernel.sem = 1250 256000 100 1024

An in /etc/security/limits:

root soft nofile 8192
root hard nofile 8192
root soft stack 20480
root hard stack 20480

Last night I scheduled some full backups an it hasn't happened any problem until now. The most weird issue is there's no error message in /nsr/logs/daemon.log.

This situation in 7.3 SP3 didn't exist, in 7.4 SP5 does.

Any idea? Have anyone installed this version in RHEL 5.3?

In my 6 years using Networker, I am very dissapointed with EMC, how could they release this buggy product?

Many thanks again.

 

[605]

Did you consider to update to 7.5.x or even 7.6?

 

[Iago77]

Really, I am thinking about it. I have found a lot of bugs in this release and My Support from EMC it seems more motivated to clase the Service Request than solving it.

Many thanks!

 

[a060463xyz]

check your paralelism:
four for every client.
12 for the client Object representing the server.
And the server paralelism to 32 or higher if you have storage node licenses.
I would upgrade to 7.6
7.4 was not the most stable release of Networker ever !

 

[Iago77]

Thanks again,

I have took a look to:

EMC NetWorker Release 7.4 and Service Packs Release Notes

And I think I have found the issue:

---
LGTsc21240
21240nw_c

Under a heavy load, index backups hang at the end of a savegroup on Linux, resulting in savegroup failure.
---

But in our version, 7.4 SP5 it seems solved when it is not.

 

[Iago77]

I have just upgraded my server to Networker 7.5 SP1 and also all the clients.

I hope this next weekend the backup works.

 

[Iago77]

It has failed again.

It is frustrating I am totally lost.

Do you have any idea about this?

 

[a060463xyz]

split the group into two or three groups with less clients per group.
Separate the start times of the groups.

 

[Iago77]

I have just modified the starttime in each groups. Also I have increased the server parallelism.

We have 3 LTO-2 and 3 LTO-3 tape drives, so it's pretty difficult adjust that parameter. I am trying using only 3 drive LTO-3. I must wait the weekend, when the full backups take place.

I will keep you up-to-date.

Many thanks for you valuable help

 

[Iago77]

Once increased server parallelism, we manage to success