Baking Clouds Ltd

Baking Clouds provide tailored IT consultancy services to small and medium-sized companies; we cover all aspects of IT without any hidden costs.

vCloud Director 9.0 error NSX Manager is unreachable.

It’s been a long time without posting a new article so here goes a new issue I found with vCloud Director 9.0.

Background

VMware Cloud stack was upgraded recently to latest version vCloud Director 9.0 build number 9.0.0.6679579.
All upgrades were completed without issues. My vCloud Director is attached to 2 vCenter Servers running v6.0U3 and v6.5.
While running health checks some errors communicating with NSX manager were detected with message “NSX Manager is unreachable”
Trying to update vCenter and NSX registration will return “Error performing operation” with error detail “Access to the specified resource has been forbidden., error code 0”

Symptoms

Log file vcloud-container-info.log in vCloud director cell shows bellow error

2017-11-03 14:56:12,460 | ERROR | VsmEventListener-192.168.1.100 | NetworkSecurityErrorHandler | Response error xml : &lt;html&gt;&lt;head&gt;&lt;title&gt;Apache Tomcat/7.0.78 - Error report&lt;/title&gt;&lt;style&gt;&lt;!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--&gt;&lt;/style&gt; &lt;/head&gt;&lt;body&gt;&lt;h1&gt;<strong>HTTP Status 403 - Account is temporary locked</strong>&lt;/h1&gt;&lt;HR size="1" noshade="noshade"&gt;&lt;p&gt;&lt;b&gt;type&lt;/b&gt; Status report&lt;/p&gt;&lt;p&gt;&lt;b&gt;message&lt;/b&gt; &lt;u&gt;Account is temporary locked&lt;/u&gt;&lt;/p&gt;&lt;p&gt;&lt;b&gt;description&lt;/b&gt; &lt;u&gt;<strong>Access to the specified resource has been forbidden</strong>.&lt;/u&gt;&lt;/p&gt;&lt;HR size="1" noshade="noshade"&gt;&lt;h3&gt;Apache Tomcat/7.0.78&lt;/h3&gt;&lt;/body&gt;&lt;/html&gt; |

&nbsp;
<p class="" style="text-align: left">2017-11-03 12:41:11,287 | WARN | pool-jetty-78 | BatchTask | Error executing backend call. | requestId=8cc0e412-2872-4b0f-a274-70b59df05a46,request=POST https://cloud.lab.local/cloud/amfsecure,requestTime=1509666070604,remoteAddress=43.241.189.206:64814,userAgent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:56.0) Gecko/...,accept=text/html application/xhtml+xml application/xml;q 0.9 */*;q 0.8 method=systemService.testShieldManagerParams
<strong>com.vmware.vcloud.api.presentation.service.BadRequestException: Failed to connect to the NSX Manager</strong></p>
at com.vmware.vcloud.net.services.vshield.impl.ShieldSessionManager.testShieldManagerConnection(ShieldSessionManager.java:275)
at com.vmware.vcloud.management.system.SystemServiceImpl.testShieldManagerParams(SystemServiceImpl.java:1314)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at com.vmware.vcloud.common.validation.AbstractMethodInterceptor.invoke(AbstractMethodInterceptor.java:42)

&nbsp;

&nbsp;

2017-11-03 12:41:11,282 | INFO | pool-jetty-78 | ShieldSessionManager | <strong>Test connection with vsm:192.168.1.233 failed to establish. | requestId=8cc0e412-2872-4b0f-a274-70b59df05a46,request=POST</strong>
com.vmware.vcloud.common.network.VsmException: Access to the specified resource has been forbidden., error code 0
at com.vmware.vcloud.fabric.nsm.error.NetworkSecurityErrorHandler.processException(NetworkSecurityErrorHandler.java:94)
at com.vmware.vcloud.fabric.nsm.error.NetworkSecurityErrorHandler.handleError(NetworkSecurityErrorHandler.java:72)
at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:700)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:653)
at com.vmware.vcloud.fabric.net.utils.impl.LoggingRestTemplate.doExecute(LoggingRestTemplate.java:64)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:613)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:531)
at com.vmware.vcloud.fabric.net.utils.impl.RestClient.getResponseEntity(RestClient.java:168)
at com.vmware.vcloud.fabric.net.utils.impl.RestClient.get(RestClient.java:138)

2017-11-03 12:40:48,932 | ERROR | pool-jetty-69 | NetworkSecurityErrorHandler | Response error xml : &lt;!DOCTYPE html&gt;&lt;html&gt;&lt;head&gt;&lt;title&gt;Apache Tomcat/8.0.44 - Error report&lt;/title&gt;&lt;style type="text/css"&gt;H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}.line {height: 1px; background-color: #525D76; border: none;}&lt;/style&gt; &lt;/head&gt;&lt;body&gt;&lt;h1&gt;HTTP Status 403 - User does not have any role on NSX Manager.&lt;/h1&gt;&lt;div class="line"&gt;&lt;/div&gt;&lt;p&gt;&lt;b&gt;type&lt;/b&gt; Status report&lt;/p&gt;&lt;p&gt;&lt;b&gt;message&lt;/b&gt; &lt;u&gt;<strong>User does not have any role on NSX Manager</strong>.&lt;/u&gt;&lt;/p&gt;&lt;p&gt;&lt;b&gt;description&lt;/b&gt; &lt;u&gt;<strong>Access to the specified resource has been forbidden</strong>.&lt;/u&gt;&lt;/p&gt;&lt;hr class="line"&gt;&lt;h3&gt;Apache Tomcat/8.0.44&lt;/h3&gt;&lt;/body&gt;&lt;/html&gt; | requestId=6bb4216a-6ee2-465c-a3de-3e8145174b96,request=POST https://cloud.lab.local/cloud/amfsecure,requestTime=1509666047430,remoteAddress=1.2.3.4:64814,userAgent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:56.0) Gecko/...,accept=text/html application/xhtml+xml application/xml;q 0.9 */*;q 0.8 method=systemService.testShieldManagerParams
Workaround
      1. Connect to NSX manager via SSH or open VM console.
      2. Log in with admin user.
      3. Type ‘enable’ to turn on privileged mode command.
      4. Type ‘configure terminal’ to enter configuration section.
      5. Run command lines to create new user and assign privileges. 
        Important note!
Password of user needs to be the same you have already configured in VCD for NSX registration. 
Later on we will update VCD DB with user name created here but password can not be changed as is not on clear text mode.

user [<em>your-nix-localuser-name]</em> password plaintext [<em>same-password-already-used-for-registration]</em> 
user [<em>your-nix-localuser-name]</em> privilege web-interface
      6. Type exit to return
      7. Type ‘write memory’ to write running configuration to memory
      8. Type ‘show running-config’ and check created user is listed
      9. In case you want to make user visible in NSX User manage on vSphere Web Client run the API calls mentioned on <a href=”https://kb.vmware.com/kb/2150736″>this KB article</a>
      10. Shutdown all VCD cells
      11. Ensure you have backups of cells and DB
      12. Change user name in vCloud DB to the new local user created in NSX. (The table name is “vshield_manager”)
      13. Wait for 5 mins to expire the account lock in NSX.
      14. Start cells one at a time.

Alternative: Wait patch release from VMware.
Update: After calling VMware support I got confirmation a new patch for vCloud Director will be released on 09/11/2017.

Other references

https://kb.vmware.com/s/article/2127351

In VMware NSX for vSphere 6.x, the admin database user is removed, and a single admin user account provides access to both the CLI and the Web-based User Interface. New installations of NSX for vSphere 6.x uses the single account approach. To help maintain backwards compatibility, when NSX Manager is upgraded from vShield Manager, NSX honors both admin accounts.

Starting with VMware NSX for vSphere 6.1.3 and later, NSX Manager authenticates using CLI credentials on the appliance management User Interface (UI) whereas the REST APIs requires a database user credentials.

vCloud Director 9.0 error NSX Manager is unreachable.
Scroll to top