JVM & Virtual Machines - Performance Considerations
If using Java in conjunction with virtual machines one needs to consider the following.
- As the number of JVM instances per VM is increased, more overhead/cost is incurred as each additional JVM is initialised on start.
- Memory in Java is managed inside the JVM and so efforts by the host virtual machine to 'optimize memory usage' by removing pages, will degrade Java application performance across the population of co-located JVM's.
If JVM's are co-located, then the required heap size for each JVM should be specified, and sufficient physical memory allocated to the underlying virtual machine to meet the total sum of JVM heap and operating system memory requirements.
To side step these considerations, Virtual Machine vendors usually suggest that instead of stacking multiple JVMs within a virtual machine, one should instead host more application instances per JVM by increasing the size of the JVM; adding more threads and enlarging the heap. However this approach simply shifts the focus of the problem from the VM to the JVM. Large heap sizes (beyond 4GB) may be required, and these may be problematic as they impact JVM performance via the increasing cost of garbage collection. Custom JVM's may be purchased with more efficient garbage collection to address the performance degradation; but this does not address the issue that multiple applications are now packed within a single failure domain.
Paremus recommend that the number of Service Fabric Fibre instances (JVM instances) per Atlas Agent (Physical / Virtual resource) should be ideally limited to 1; but no more than 4. If this upper limited needs to be exceeded your requirements should be discussed with Paremus.
'GConf ...' D-BUS Library (RedHat 6)
The Linux package
dbus-1.2.24-7.el6_3.x86_64 used by desktop applications such as
EMACS is know to cause the JVM to crash on RedHat 6 versions of Linux. This package should be uninstalled and servers being used as Service Fabric Fibres.
Note: No longer seems the be a problem when using Java 8.
Use of 127.0.1.1 Loopback (Debian/Ubuntu)
Using the 127.0.1.1 loopback address bound to the
/etc/hostname with cause Service Fabric Fibres to fail to communicate with each other.
/etc/hostname must be bound to the public network interface that you expect the Service Fabric fibre to communicate over.
Ensure Java is installed on the nodes and
The Service Fabric expects Java to be either on the
PATH, or specified by the
JAVA_HOME environment variable. Typically these are either set system-wide in
/etc/bashrc, or set by an individual user in
Atlas is invoked by
/etc/init.d/atlas using '
su -c paremus .... ', so it should inherit the environment of the '
paremus ' user. Alternatively, you can set
If a change is made to
atlas_env.sh, Atlas must be restarted for the new environment to take effect.
'libgcc_s.so.1 => not found' – or equivalent errors relating to missing 32-blt libraries.
Check missing Atlas dependencies on the Unix platform
Install missing library
I do not have remote display access to the node to which I intend to install the Service Fabric. Given that I cannot run lzPack on the node, how should I install the Service Fabric software?
The lzPack has the option of creating an install script which may be used to install the Service Fabric to a compute node using the command line; see headless installation.
If no initial peers are set, the Paremus
Service Fabric will attempt to use IP multicast to discover
By default the discovery process uses the following IP address and ports:
184.108.40.206 jini-announcement - port 4160 220.127.116.11 jini-request - port 4160
If you intend to use Multicast and find that a subset of your fibre population is unable to join a
Service Fabric, check that multicast is configured appropriately.
Also note; when using a single machine there are configurations in which multicast packets are not propagated between processes.
click to download ) is a lightweight command line tool that can be used to provide initial diagnostics of network & host support for IP multicast. nicnack is network interface aware, i.e. it provides separate information for each network interface.
nicnack may run in one of three modes:
This just lists all network interfaces on the machine, along with ip and host name information. For example:
In this mode nicnack multicasts a custom message every two seconds to a specified multicast group and port. This can be read by other nicnackinstances in receive mode.
In this mode nicknack listens for and displays messages sent to a specified multicast group and port by nicknack instances in send mode. Used in conjunction with Send mode this can be used to establish whether or not multicast traffic is successfully propagated between two hosts. Note that multicast visibility is not symmetric, i.e. host A's ability to send multicast packets to host B does not imply host B's ability to send packets to host A. A sample from a receive mode nicknack session follows.
If, having run the nicknack utility, you suspect that IP multicast may be the issue, the following two areas should be looked at in more detail.
The security firewall on one, a subset, or all of your machines that are running the Paremus Service Fabric environment may be configured by default to block IP multicast traffic.
- Linux - To enable multicast send / receive capability for Linux systems, insert the following entry into the operating system's iptable
INPUT -d 18.104.22.168/4 -j ACCEPT
- Windows - In the case of Windows XP, by default, the Group Policy settings for the Windows Firewall are "Not Configured" for all objects. This allows the Windows Firewall to use its default settings, which are quite restrictive. With respect to multicast the default settings prohibits unicast response to multicast or broadcast requests.
On the relevant machines, edit Network > Network Connections > Firewall and set a disable policy from the following options.
For other Firewall products or alternative Microsoft operating system versions. Check relevant documentation.
Simple layer II network switches treat multicast traffic in the same manner as broadcast traffic, that is, they will forwarded multicast packets to all active switch ports. If your Paremus Service Fabric test machines connect to a layer II network, comprising of one or more simple layer II ethernet switches (these interconnected without intervening layer III routers), then the network is unlikely to be the cause of IP multicast connectivity issues.
In more sophisticated environments, network infrastructure supports a mulitcast protocol known as IGMP. Within an IGMP enabled network environment, traffic associated with a multicast group is only forwarded to ports that have members participating in that group. A layer-2 switch supporting IGMP snooping can passively snoop on IGMP Query, Report and Leave (IGMP version 2) packets transferred between IP Multicast Routers/Switches and IP Multicast hosts (on each switch port), to learn the required IP Multicast group membership required by each port. The advantage of using IGMP snooping is that it generates no additional network traffic, whilst significantly reduce multicast traffic passing through your switch - as all multicast is only targeted to hosts that have registered interest in the multicast group.
If Paremus Service Fabric functions correctly when run on a single machine and also when run in a distributed environment with machines connected via a simple layer II network, but fails in a more complex network environments, then multicast configuration of the network is the the most likely cause.
In such circumstances politely explain the problem to your network administrators. The network administrators will be able to help you diagnose the issue in greater detail, and may be willing/able to disable IGMP snooping on the relevant switches to verify whether or not IGMP is a contributing factor.
Usually the Service Fabric license is installed during the lzPack installation process process. If for some reason the appropriate
license.ini file is not available at installation time, it can be subsequently copied into the
$install_root/etc directory. The Fibre image should then be re-built as explained here. The same process is used to update an expired license.
How do I request multiple instances of a Managed Service Factory part?
Managed Service Factory (MSF) parts can be created as follows:
In this example, the
name attribute specifies both the name of the element within the document and the Persistent Identity (PID) of the configuration record.
In order to create multiple configuration records for the same PID, it is necessary to create two parts with different names, and override the name/PID mapping. This can be done by setting the
part attribute as follows:
Note that the
name attribute is different, in order two create two distinct records, but both records have the same
part value and therefore both map to the configuration PID
Cannot connect to Fabric Management
If the redirector fails to locate Entire, it will print an error message: "Unable to redirect". This could happen if:
- There is no Infra Fibre currently running in the Fabric.
- Or a network issue prevents the selected Fibre from seeing the active Infra Fibre(s).
Repositories blocked by Proxy
When using Atlas place the equivalent of the following in the
$FABRICHOME/var/atlas start scripts.
Alternatively if you are using an
Environment configuration file use the following JVM start flag.
Viewing Port Usage
View ports currently in use by a Fibre, using the Unix
Setting Port Ranges
The base.Port is value is defined in /etc/posh/environment. Each Fibre uses a default port range starting at the base port and then incremented when each additional Fibre is started on the same host: (9000 + (100 * fibre instance-id).
This value is then used as follows in
locationPortparameter is used by the manage:detect command;
locationPortis not offset by
instance-id, as when multiple Nimble nodes are running on the same host, only one of them is required to use the
maxPortcontrol the range of dynamically allocated ports for each Fibre.
httpPortis the default value used by an HTTP service if not overridden by the OSGi Configuration Admin (usually in
The default is to assign dynamic ports in the range
9000 -> 9099. However, this can be limited to a smaller range.
One may run the risk of port starvation if a too restrictive port range is used and too many services are installed into the same Fibre. Before modifying these values changes should be discussed with your Paremus Support Engineer.