HCX Security Information
Executive Summary
VMware HCX is a workload migration and mobility solution that provides management, orchestration, and observability for migrating and rebalancing workloads across multi-cloud environments. With HCX, customers will benefit from visibility into multi-site connectivity and migration activities, helping them accelerate their large-scale workload migration projects seamlessly, securely and with minimal to no downtime.
Due to the critical role VMware HCX plays within an organization, configuration of the product along with secure technology will reduce the risk that an organization may encounter in a multi-cloud environment. The objective of this document is to provide information and topology recommendations to ensure secure deployment of VMware HCX. To ensure that your system is securely hardened, review the recommendations, and assess them against your organization's security policies and risk exposure.
HCX Architecture
The main components of VMware HCX are HCX Manager (i.e., Connector or Cloud depending on the role it plays), HCX Interconnect (HCX-IX), HCX Network Extension (HCX-NE), HCX WAN Optimization (HCX-WO), and HCX Sentinel Data Receiver and HCX Sentinel Gateway (HCX-SDR/SGW: OSAM Sentinels). Adequate care must be undertaken towards the placement and connectivity of these components within an organization or across a multi-cloud environment. HCX functions can be divided into two categories: management plane and data plane. HCX Manager provides management plane functionality.
HCX Management Plane
Consumption of HCX Manager can be driven directly via HCX UI/CLI or from vCenter through HCX plug-in. Access to HCX Manager UI is only provided via HTTPS with TLS 1.2. CLI access is provided securely via SSH. HCX Manager integrates with vCenter and NSX Manager and they communicate over the encrypted tunnels. HCX Managers (i.e., cloud and Connector) establish the secure connection (i.e., TLS 1.2) needed for management, authentication, and orchestration of HCX services between source and destination.
HCX Data Plane
The HCX data plane is implemented via HCX Service Mesh or Fleet Appliance –HCX-IX, HCX-NE, HCX-WO, and HCX-SDR/SGW. These appliances are deployed symmetrically in source and destination locations. HCX-IX and HCX-NE components automatically establish encrypted tunnels (i.e., IPsec) with the corresponding peers across source and destination sites to ensure secure communication between them. These are unmanaged encrypted tunnels. HCX-WO and HCX-SDR/SGW connect to their respective peers between source and destination locations through the HCX-IX appliances (i.e., over the encrypted tunnel).
HCX Deployment – Port and Protocol Requirements
Different HCX components communicate with one another to provide a scalable distributed workload migration and balancing platform. The set of TCP/UDP ports used between different components of HCX are documented in the release specific documents. The following ports must be allowed in HCX Manager deployments:
• The source site firewalls must be configured to allow outbound connections to the destination site systems.
• The destination site firewalls must be configured to allow inbound connections from the source site systems.
• Connections initiated from the source to the destination site.
• Connections within a single site, either at the source or the destination environment. These connections never traverse from the source to the destination or from the destination to the source.
• Connections made when the HCX is added as a solution in a VMware Aria Operations installation.
For a complete list of network port and protocol requirements, see VMware Ports and Protocols.
HCX Manager Deployment
As HCX Manager is part of the management plane, certain considerations must be undertaken as to placement and connection.
Placement, Physical and Network Security
Best practice dictates that HCX managers are deployed in a segmented and secure network. Typically, HCX Managers are deployed in the vSphere management network along with other HCX components, vCenter, and NSX Manager. HCX Managers need to communicate with other HCX components, vCenter, and NSX Manager. For added security, you can provide additional isolation by deploying HCX Manager in a separate network segment and protected via Firewall.
Access and Login
Login to the HCX Manager can be done via Manager UI or Appliance Management UI over HTTPS, vSphere Client by vSphere Client Plug-in, and HCX Manager CLI by using VMRC or SSH. Please refer to the HCX User Guide for details.
NTP
HCX requires a valid NTP server synchronized time for integrated systems operations, such as accurate audit and log keeping. The system asks for initial NTP Server settings during the OVA deployment in the vCenter Server. You must configure a trusted remote NTP server for time synchronization. The NTP server must be an authoritative time server or at least synchronized with an authoritative time server. Do not use host time or any other non-authoritative time source.
Syslog
Within the HCX Manager, up to two syslog servers can be configured to send log messages to a remote location. For details, see HCX User Guide.
SSH
The Secure Socket Shell (SSH) service is activated by default during the initial HCX Manager installation. SSH can be disabled or enabled from the HCX Appliance Management Interface.
SSL Certificates
HCX appliances generate self-signed certificates during installation. These certificates are used to access HCX services and appliances through one of several interfaces:
vSphere Client – Utilizes a vSphere Client Plug-in to manage all HCX services.
HCX Manager UI - Logging in to the HCX Manager at https://hcx-ip-or-fqdn:443 to manage all services.
HCX Appliance Management - System management, licensing, and upgrade operations are performed by logging in to the HCX Manager at https://hcx-ip-or-fqdn:9443.
Login Password
To login into HCX Manager, users must provide a “password” created during the installation. It is recommended to change login password frequently based on the organization’s security standard.
Users and Roles
HCX utilizes various accounts and roles in deploying, managing, and operating the system. As a security best practice, you must secure the VMware HCX console and manage Secure Shell (SSH), administrative accounts, and console access.
HCX has the following account requirements:
| Account | Requirements | Additional Information |
| admin | The admin password must be set. The root password must be set. | Created during the OVA deployment. Used in the Appliance Management (https://hcx-ip-or-fqdn:9443) Used for CLI/terminal shell access. |
| Account for vCenter Server Registration | The account must belong to the vSphere administrators’ group or have the administrator role assigned. | The administrator@vsphere.local account is suggested by default, but not required. Alternate vSphere SSO local users that meet the requirements can be used. Active Directory service accounts that meet the requirements can be used. See Appendix A for the minimum vSphere Privileges required for an HCX vCenter Registration Account. |
| Account for NSX Registration | If NSX-T, this account must have the Enterprise Admin role assigned. If NSXv, this account must have the Enterprise Administrator role assigned. | The NSX admin account is suggested by default, but not required. Alternate NSX local accounts that meet the requirements can be used. Active Directory service accounts that meet the requirements can be used. Prior to NSX-T Data Center 3.0, it was mandatory to use the NSX admin account. Note: This account is generally not required for HCX Connector installations. It is required only when extending NSX Segments or migrating NSX Tags. |
| Account for vCloud Director Registration | The account must have the System Administrator role assigned. | The VMware Cloud Director’s sysadmin account is suggested by default, but not required. An alternate local account that meets the requirements can be used. LDAP service accounts that meet the requirements can be used. Note: This account is only required for provider installations of VMware HCX with VMware Cloud Director. A tenant does not require this account. |
| Accounts for HCX Role Mapping (This refers to SSO User accounts that will be mapped to an HCX role.) | The user’s group must be included in the HCX Role Mapping configuration. | HCX supports two user roles: Administrator and Tenant: HCX Administrator is for those who configure and operate HCX (create and manage Compute Profiles, Site Pairings, Service Meshes, Network Extensions, Migrations, and DR operations). HCX Tenants are for Service Provider installations only. This role does not support adding or deleting Network Profiles. The vsphere.local\Administrators vSphere SSO Group is added by default to HCX Administrator. However, it is not mandatory to use this SSO group. For the HCX Tenant role, no default group is provided. A common practice is to create an hcx-administrators vSphere SSO Group. SSO and Active Directory users are populated into the hcx-administrators vSphere SSO group. The default vsphere.local\Administrators HCX Administrator user group entry in the HCX Role Mapping configuration is replaced with the new hcx-administrators vSphere SSO group. |
| Site Pairing Accounts | The user’s group must be included in the HCX Role Mapping configuration (on the remote HCX Cloud system being paired). The user's group can be in either the HCX Administrators group or the HCX Tenant group. | The site pairing user is entered along with the HCX Cloud’s URL in the site pairing configuration on the source HCX Manager system. The following are typical scenarios: In a private data center HCX deployment, the site pairing user is traditionally the administrate user for the destination vSphere environment. In a dedicated public cloud HCX deployment, the site pairing user is traditionally the SDDC administrator account provided to the tenant. In a VMware Cloud Director HCX deployment, the site pairing user is the Organization Administrator account. |
Note: vCenter Server, NSX Manager, and VMware Cloud Director registration accounts (“service accounts”) must have global object access.
Role Mapping SSO Users
Access to HCX services and features depends on the assigned user role. User roles are assigned in the HCX appliance management interface during the initial HCX activation and configuration. Currently, there are only two user roles that can be assigned.
HCX Administrator - SSO groups assigned to the HCX Administrator role have unrestricted access to perform all HCX configurations and operations.
HCX Tenant - This role is intended for use by HCX Service Providers and is not available in a HCX Connector deployment. SSO groups assigned to the HCX Tenant role cannot add or delete HCX Network Profiles.
HCX Certificates and their usage
HCX managed sites connect with each other over HTTPS. During the site pairing process, the service mesh checks the site certificates before making the secure connection between sites. For the connections to succeed, the site pair must know the TLS end-point certificate. For self-signed certificates, or any type of certificate that is not universally known, you must add that certificate to the site.
HCX currently allows you to add or remove certificates from the system certificate store or update the local server certificate for appliance management. Note that modifying the certificates in use or deleting non-user imported certificates can impact signing chains or otherwise impact HCX appliances, including preventing or disrupting site pairing or other communication.
Only User Interface connections for the Hybridity Manager URL on TCP 443 can be replaced by CA signed certificates. Appliances will also use TCP 9443 for data plane communications and for individual appliance management; this certificate cannot be replaced or deleted.
If replacing self-signed certificates with commercial or local certificate authority generated keys, the HCX Manager and Connector Appliances must contain and/or trust all authorities in the signing chain.
HCX Alerts and Logs
Alerts
HCX generates alerts with different severity levels to flag events such as migration, network stretch failure, or reachability issues connecting to an endpoint or paired site. Alerts are available via the HCX Dashboard or the Migration tab in the vSphere Client.
Within the HCX Dashboard, alerts can be Acknowledged, Reset to Green, or Suppressed. The actions available depend on the type of alert that is generated. Types of alerts are Critical, Warning and Informational.
Within vCenter, HCX creates some default vCenter Alarms that are generated when a migration fails. By default, these alarms trigger a UI indication using the red critical icon on the VM involved.
Because these are vCenter Alarms, you also have the option of creating additional actions such as sending an email alert, an SNMP trap or run a command within vCenter Server. The default Alarms can be updated to include these actions, or you can use the EventID generated by HCX Events to create new or additional vCenter Alarms.
Additional information on creating vSphere alarms can be found in VMware Docs for vSphere here. Appendix X contains a list of the HCX EventIDs that can be used with Alarms.
Logs
HCX logs can be found in different locations depending on the components generating the logs. HCX logs are stored on the appliance disks. The logs are stored in plain text but are accessible only by a user logged in via SSH. Log bundles can be generated from the command line for download and analysis or to provide to VMware Support.
On the Management Appliance, they can be accessed from /common/logs/admin. From within this directory, you can open the relevant logs using standard Linux text commands.
The following logs are available on the HCX Manager:
App.log – Logs all activities for the App Engine service.
Job.log – Logs all job related tasks and information.
Web.log – Logs all activities for the Web Engine service.
/audit – This directory contains the appliance audit logs. These are also accessible from the HCX Admin UI.
/common/logs/admin/audit_logs– This directory contains the appliance audit logs. These are also accessible from the HCX Admin UI.
/common/logs/upgrade/upgrade.log : All upgrade tasks and related information are logged here.
/common/logs/appliance-management/appliance-management.log: All appliance management related tasks and information are logged here.
/common/logs/kafka/: All Kafka Server related tasks and information are logged here.
/common/logs/zookeeper/: All Zookeeper related tasks and information are logged here.
/common/logs/postgres/: All PostgresDB related tasks and information like the postgres SQL queries taking longer time than usual are logged here.
/common/logs/backup/: HCX Manager’s backup log is available here and available only when the backup operation has been initiated at least once.
/common/logs/restore/: HCX Manager’s restore log is available here and available only when the restore operation has been initiated at least once.
Note: Use VMRC or SSH to connect to the HCX Manager shell. Switch user to root: su –.
Change directory to /common/logs/admin. Within this directory, you can open the relevant logs using standard Linux text commands.
On the HCX Interconnect Appliance, logs can be accessed from /var/log/vmware.
The following logs are available on the HCX Interconnect Appliance: Hbrsrv.log
Hbrsrv.log: vSphere Replication tasks and related information are logged here.
MobilityAgent.log: vMotion tasks and related information are logged here.
/var/log/messages: System kernel log messages as well as information related to tunnel status can be found here.
Logging Levels
HCX Manager logs are available at different log levels: [Info],[Warning], [Error] and [Debug]. Debug level logs can be achieved by configuring it at the below location:
/opt/vmware/config/app-logback.xml
/opt/vmware/config/web-logback.xml
/opt/vmware/config/appl-mgmt-logback.xml
The debug level can be configured at both the root level as well as a service specific level as below:
Root Level:
<root level="DEBUG">
<appender-ref ref="APP_FILE"/>
<appender-ref ref="JOB_FILE"/>
<appender-ref ref="SYSLOG"/>
</root>
Service Specific Level:
<logger name="com.vmware.hybridity.service.vmotion" additivity="false" level="DEBUG">
<appender-ref ref="APP_FILE"/>
<appender-ref ref="JOB_FILE"/>
<appender-ref ref="SYSLOG"/>
</logger>
Log Format
HCX Syslogs are included in vRealize Log Insight (vRLI) and follow the RFC 5424 format. Below are some of the information related to filters to be applied in vRLI in order to search for HCX logs:
1) component: hcx , text: hcx-app , text: level=info
2) component: hcx , text: hcx-app , text: level=error
3) component: hcx , text: hcx-web , text: level=info
4) component: hcx , text: hcx-web , text: level=error
5) component: hcx , text: hcx-appl_mgmt , text: level=info
6) component: hcx , text: hcx-appl_mgmt , text: level=error
Additionally, we also have audit logs with the component names as below:
hcx-app-audit
hcx-web-audit
hcx-appl_mgmt-audit
Here is a Sample of the HCX logs as observed from a vRLI instance:
<99>1 2023-06-15T14:23:27.667+00:00 hcx.sddc-54-214-58-196.vmwarevmc.com hcx-web-audit - - - [https-jsse-nio-8443-exec-10] [LEVEL=INFO] c.v.vchs.hybridity.audit.AuditTrail {"internal":{"threadEnterprise":"Ent: HybridityAdmin","threadUser":"No Thread Context","thread":"https-jsse-nio-8443-exec-10","lineNumber":310,"classname":"com.vmware.vchs.hybridity.authentication.AuthenticationFilter","method":"doFilter"},"userIdentities":[{"username":"Administrator@VMC.LOCAL","enterprise":"HybridityAdmin","organization":"DEFAULT","userRoles":["System Administrator"],"endpointId":"20230609212947432-49cd1944-1f92-47e8-ba52-1ff03f8cfd76"}],"userIdentity":{"username":"Administrator@VMC.LOCAL "},"eventId":"c431d057-db86-42e5-a2e3-66bb58ab7d4f","eventTime":1686839007665,"message":"Authentication success","restEndpoint":{"uri":"\/hybridity\/api\/alerts?action=query","sourceIPAddress":"127.0.0.1"},"requestParameters":{"query":[{"action":["query"]}],"sessionId":"271d64ff8a402e297cfb4ff0e207fb7b03db447f0043577a1d68b8ed7753f068"},"operationalStatus":"NOT_APPLICABLE"}
HCX Tech Support Log Bundle Generation
Log bundles can be generated from both Appliance Management Interface (port 9443) and HCX Service UI (port 443) for download and analysis or to provide to VMware Support. It is recommended to collect Core HCX Logs, Database Dump, and Interconnect Appliances logs (IX/NE) by selecting the options as applicable.
HCX Service UI (Troubleshooting Tab):
HCX Appliance Management UI (Troubleshooting Tab)
CLI to generate HCX Logs
From the upcoming HCX 4.8.0 release, there is a Utility script specifically to generate the logs when the web interface becomes unavailable, for example if some of the critical services in the appliance stop or fail to start. Here is the snapshot of the command to be used after having performed SSH to the HCX Manager:
Built In HCX Capabilities for Security Compliance
This section summarizes some of the key built-in HCX platform security related capabilities to make HCX a more secure system and to achieve the security compliance requirements:
HCX uses self-signed certificates for the Appliance Management Interface (port 9443) and HCX Service UI (port 443) appliances.
HCX Management remote SSH uses a FIPS-validated cryptographic module.
HCX components internal communications are encrypted via TLS 1.2.
HCX Manager’s communication with vCenter and NSX are encrypted via TLS 1.2.
HCX components identification and authentication is done via self-signed certificates. This enables initial trust between HCX components and prevents a rogue device from asserting the identity of any HCX components.
HCX enforces minimum password complexity.
HCX Manager has the option to use valid certificates from a trusted authority.
HCX provides remote logging by forwarding logs to a centralized storage. This mitigates risks of local loss of logs and enabled central system to easily monitor logs.
HCX ensures that users are not able to access the database directly. Database access is allowed only via HCX appliance.
HCX locks out users when a user enters a wrong password multiple times. This helps to prevent DDOS and brute force attacks into the system.
HCX ensures automatic SSH session timeout and logs the user out if the user is inactive for more than a certain amount of time.
HCX Manager UI is reachable only via port 443 and 9443. Only these ports need to be open for management operations.
HCX automatically terminates a user session after user session timeout.
HCX does not store the password of other systems rather it communicates via an internal system API call to login.
As outlined in the HCX Data Privacy Statement, HCX does not store any sensitive data on the appliance or exchange with connect.hcx.vmware.com.
VMware Security Development Lifecycle, Policies, and Advisories
As part of HCX release cycle, HCX components go through the penetration and vulnerability-scan test to harden appliances. More info on VMware Security development is covered in the following white paper – VMware Security Hardening Activity.
Here are the other links on VMware Trust & Assurance and Security Response Policies. Additionally, VMware regularly publishes Security Advisories with information on what VMware products are affected by known threats.
| Author | Rabiul Hasan |
| Co-Author | Sailesh Jha |
| Co-Author | Gabe Rosas |