VMware Backup Failure with NetBackup 7.5, Transport Mode Error 23, Status 6

I thought I would post a solution here to a Status 23 error from bpbrm which results in a Status 6 on the parent VMware backup job (child job never spawns).

The particulars:

NBU version: 7.5.0.1 / 7.5.0.4
Master Server: Solaris 10 (SPARC)
Media / Backup Host: Windows 2008 R2

ESXi 4.1.0 / vCenter 4.1.0
Storage: NetApp 3240

Policy Type: VMware
Storage Policy: Local DSSU
BLIB: Yes
Transport mode: NBD/NBDSSL (this issue can also apply to SAN)

If you're like me, you've already looked into the possible solutions below...

Ensure hostname resolution works for all ESXi hosts / vCenter to/from the Master Server
https://www-secure.symantec.com/connect/forums/error-opening-snapshot-disks-using-given-transport-mode-status-23-backup-failed-back-requeste
https://www-secure.symantec.com/connect/forums/netbackup-vmware-error-opening-snapshot-disks-using-given-transport-mode-status-23

Are the datastores presented/accessible to the VMware Backup Host? (not applicable for NBD)
https://www-secure.symantec.com/connect/forums/status-code-6-and-23-vstorage-api
https://www-secure.symantec.com/connect/forums/error-opening-snapshot-disks-using-given-transport-mode-status-23

...but to no avail. Engaged tech support, turn VxMS logging on etc.

Logs produced look similar to this:

Detailed Status (from Activity Monitor):
10/23/2012 10:45:07 PM - Error bpbrm(pid=8764) from client <hostname>: ERR - Error opening the snapshot disks using given transport mode: Status 23
10/23/2012 10:45:08 PM - Info bptm(pid=4724) waited for full buffer 1 times, delayed 1619 times  
10/23/2012 10:45:08 PM - Info bptm(pid=4724) EXITING with status 90 <----------      
10/23/2012 10:45:08 PM - Info bpbkar32(pid=5064) bpbkar waited 0 times for empty buffer, delayed 0 times.  
10/23/2012 10:45:11 PM - Error bpbrm(pid=8764) could not send server status message     
10/23/2012 10:45:11 PM - Critical bpbrm(pid=8764) unexpected termination of client <hostname>      
the backup failed to back up the requested files(6)

bpfis Log:
09:53:23.402 [6844.7840] <2> onlfi_vfms_logf: INF - vfm_fi_get_credentials: No disk array credentials found on server
09:53:23.621 [6844.7840] <2> onlfi_vfms_logf: INF - vfm_fi_get_credentials: No storage server credentials found on server
09:53:37.521 [6844.7840] <2> onlfi_vfms_logf: INF - vmwareLogger: RetrieveHostContents: Datastore: name = vmbackup is NOT accessible
09:53:37.521 [6844.7840] <2> onlfi_vfms_logf: INF - vmwareLogger: RetrieveDatastoreContents: datastore vmbackup is NOT accessible
 

VxMS Log:
18:57:28.0408 : connectToHost:xGuest.cpp:1017 <INFO> : TransportMode requested: nbd
18:57:28.0408 : connectToHost:xGuest.cpp:1022 <INFO> : Transport Modes available on this host: file:san:hotadd:nbdssl:nbd
18:57:30.0265 : vdConnect:xInterface.cpp:869 <DEBUG> : Done with VixDiskLib_Connect(): 214236752
18:57:30.0265 : vdOpen:xInterface.cpp:367 <INFO> : Calling VixDiskLib_Open()
18:57:30.0265 : g_vixInterfaceLogger:bvix.cpp:1833 <INFO> : VixDiskLibVimResolveHostName: Resolving IP address for hostname server.domain.com.
18:57:30.0265 : g_vixInterfaceLogger:bvix.cpp:1833 <INFO> : VixDiskLibVimResolveHostName: Resolved to 10.10.10.10.
18:57:30.0280 : g_vixInterfaceLogger:bvix.cpp:1833 <INFO> : VixDiskLibVim: VixDiskLibVimLogin
18:57:31.0185 : g_vixInterfaceLogger:bvix.cpp:1837 <WARN> : VixDiskLibVim: Not licensed to use this function.

This error will repeat for all available transport modes.

Steps we tried that did not work:

• Removed and re-added client to policy
• Upgraded from 7.5.0.1 to 7.5.0.4 (a suggestion from Symantec)
• vMotion VM to another ESXi host
• Storage vMotion VM to another datastore

Solution:

We cloned the VM and the clone successfully backed up. Which led us to wonder if it was a config issue on the VMware side (VMX file?).

After taking an outage for the client we did the following:

1. Remove the offending VM from vCenter inventory.
2. Rename the VM folder using the datastore browser. (This step is to avoid the same name folder that will be created in step 3)
3. Create a new VM with the same name and specs. But no disks attached.
4. Move the vmdk files from the old file folder to new folder.
5. "Edit settings" on the newly created VM and attach "existing disk" by locating the VMDK files.
6. Power on the new VM.
7. Remove the old (renamed) file folder which should contain only the vmx, nvram and log files [house cleaning]

The client was then able to backup successfully.