JAVA Mailer not Starting

Today was the bad day for us, we had an outage last weekend and after we finished our job lot of things start failing. Due to DBA errors and Java mailer is one off them. Here is what log says
------------------------------------------------------------------------------------
SVC-GSM-WFMLRSVC-67585-10006 : oracle.apps.fnd.cp.gsc.Logger.Logger(String, int) : Logging to System.out until necessary parameters are retrieved for Logger to be properly started.oracle.apps.fnd.cp.gsc.SvcComponentContainerException: Could not start component; performing rollback -> oracle.apps.fnd.cp.gsc.SvcComponentException: Validation failed for the following parameters -> {FRAMEWORK_USER=Unable to make a connection with the user and responsibility}. Parameters were -> {INBOUND_MAX_IGNORE_SIZE=1000, INBOUND_MAX_LOOKUP_CACHE_SIZE=100, ATTACH_IMAGES=N, ALLOW_FORWARDED_RESPONSE=Y, INBOUND_UNSOLICITED_THRESHOLD=2, NODENAME=PRODMAIL, SEND_UNSOLICITED_WARNING=N, AUTOCLOSE_FYI=N, PROCESSOR_READ_TIMEOUT_CLOSE=Y, INBOUND_PROTOCOL=IMAP, ATTACHED_URLS=WFMAIL:ATTACHED_URLS, TEST_ADDRESS=, EXPUNGE_ON_CLOSE=Y, PROCESSOR_OUT_THREAD_COUNT=1, ATTACH_STYLESHEET=N, OUTBOUND_CONNECTION_TIMEOUT=120, INBOUND_MAX_RET_EMAIL_SIZE=100, MAX_INVALID_ADDR_LIST_SIZE=100, PROCESSOR_MAX_LOOP_SLEEP=60, SEND_ACCESS_KEY=N, PROCESSOR_IN_THREAD_COUNT=1, FRAMEWORK_APP=1, CLOSED=WFMAIL:CLOSED, INBOUND_FETCH_SIZE=100, SUMMARY=WFMAIL:SUMMARY, ENABLE_STYLESHEET=N, PROCESSOR_ERROR_LOOP_SLEEP=60, OPEN_MAIL_FYI=WFMAIL:OPEN_MAIL_FYI, FRAMEWORK_RESP=20420, ALTERNATE_EMAIL_PARSER=oracle.apps.fnd.wf.mailer.DirectEmailParser, INBOUND_PASSWORD=_@8@!864^4$9#64#^@#8@@686##!#44@#!0!99@@`9+^+*#B$$@99*$, HTTP_USER_AGENT=Mozilla/4.76, RESET_NLS=N, PROCESS=Process, OPEN_MAIL_DIRECT=WFMAIL:OPEN_MAIL_DIRECT, PROCESSOR_LOOP_SLEEP=5, MESSAGE_FORMATTER=oracle.apps.fnd.wf.mailer.NotificationFormatter, REPLYTO=emailuser@edomain.com, FRAMEWORK_USER=0, CANCELED=WFMAIL:CANCELED, OUTBOUND_PROTOCOL=SMTP, ACCOUNT=emailuser, DISCARD=Discard, FROM=Expedite, PROCESSOR_DEFER_EVTDATA_READ=Y, SEND_CANCELED_EMAIL=N, WARNING=WFMAIL:WARNING, PROCESSOR_MAX_ERROR_COUNT=50, INBOUND_CONNECTION_TIMEOUT=120, HTMLAGENT=http://hostname.domainname.com:8000/pls/SID, INBOX=INBOX, OPEN_INVALID_MORE_INFO=WFMAIL:OPEN_INVALID_MORE_INFO, OUTBOUND_SSL_ENABLED=N, INBOUND_SERVER=mail.domain.com, OPEN_MORE_INFO=WFMAIL:OPEN_MORE_INFO, MAILER_SSL_TRUSTSTORE=NONE, INLINE_ATTACHMENT=N, OPEN_INVALID=WFMAIL:OPEN_INVALID, EMAIL_PARSER=oracle.apps.fnd.wf.mailer.TemplatedEmailParser, DIRECT_RESPONSE=N, OPEN_MAIL=WFMAIL:OPEN_MAIL, FRAMEWORK_URL_TIMEOUT=30, COMPONENT_LOG_LEVEL=1, SUMHTML=WFMAIL:SUMHTML, PROCESSOR_READ_TIMEOUT=10, DEBUG_MAIL_SESSION=N, INBOUND_SSL_ENABLED=N, OUTBOUND_SERVER=mailhost.domain.com}
at oracle.apps.fnd.cp.gsc.SvcComponentContainer.handleComponentEvent(SvcComponentContainer.java:2211)
at oracle.apps.fnd.cp.gsc.SvcComponentContainer.onBusinessEvent(SvcComponentContainer.java:300)
at oracle.apps.fnd.wf.bes.DispatchThread.run(DispatchThread.java:57)
----------------------------------------------------
And I think I know the issue, it was because of autoconfig only, you might be asking why? Our DBA ran autconfig without clearing FND_Nodes table. And as we were running on virtual host (clustering at db tier). It created a mess there. We have duplicate server_id for virtual and active physical host. Server ID for both of them were same, that's the reason Java mailer was not able to start and failing because of above error. Now what, another downtime---- I don't think customer will agree, but this issue is critical how to resolve this without big downtimes for running autoconfig. We raised Sev 1 tar to see if Oracle Support can help on this, but hard luck, they said the same old story of running autoconfig and all. But after spending few minutes, I got the solutions with 10 min of downtime. Here is what I did :

1. Shutdown Concurrent Manager
2. Take a backup of FND_Nodes.
3. Delete Physical host from OAM (OAM > Administrator > Hosts)
4. Register same host again from OAM.
5. Start Concurrent Manager
6. Mailer was working fine after this.

In your case you need to re run autoconfig, but prior to that run this command as apps user.

EXEC FND_CONC_CLONE.SETUP_CLEAN;
Commit;

Oracle Apps Free Lance