Changing the heap size of Enterprise Manager 12c

Enterprise Manager 12c is a product that is being deployed a lot lately, a vast improvement over previews version. For most DBA’s it is a tool they use regularly on a daily basis so performance is key. Fortunately oracle provides us with some documentation on how to size the repository database and heap size of the Oracle Management Service. An excellent white paper on EM12c sizing can be found here on OTN:

Click to access em12c-sizing-1590739.pdf

Interestingly enough it states that Oracle considers installation with +1000 targets as a medium sized em12c environment. That limit of +1000 targets is reached quite easily. One of the advisories is that for a medium EM12c environment the minimum heap size should be at least 4G, most default installation run with a heap size of around 1.5G.

So how do we change the heap size of OMS to 4G, meet the script, this script can be located Grid Control domain directory of Weblogic:

[oracle@em12c ~]$ cd `cat ./bea/beahomelist|cut -d\; -f2`/gc_inst/user_projects/domains/GCDomain/bin
if [ "${SERVER_NAME}" != "EMGC_ADMINSERVER" ] ; then
  USER_MEM_ARGS="-Xms2048m -Xmx4096m -XX:MaxPermSize=1500M -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing -XX:ReservedCodeCacheSize=100M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled"
  if [ "${JAVA_VENDOR}" = "Sun" ] ; then
    if [ "${PRODUCTION_MODE}" = "" ] ; then
      USER_MEM_ARGS="-Xms2048m -Xmx4096m -XX:MaxPermSize=1500M -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing -XX:ReservedCodeCacheSize=100M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:CompileThreshold=8000 -XX:PermSize=128m"
  export USER_MEM_ARGS

Change the Xms and Xmx parameters on line 2 and 5 the desired values, in this example i choose to set a maximum heapsize of 4G and minimun if 2G. No smart caclculations are done to determine these values, except the upper limit, which came from the earlier mentioned white paper.

[oracle@ovmem01 bin]$ . ./
[oracle@ovmem01 ~]$ $OMS_HOME/bin/emctl status oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Up
[oracle@ovmem01 ~]$ $OMS_HOME/bin/emctl stop oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
[oracle@ovmem01 ~]$ $OMS_HOME/bin/emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
[oracle@ovmem01 ~]$

That is about it about changing your heap size for EM12c, short but hopefully interesting post.

Adding an Exadata V2 as a target in Enterprise Manager 12c

Although Oracle says that with Enterprise Manager 12c it “provides the tools to effectively and efficiently manage your Oracle Exadata Database Machine” it is a bit of a challenge to get it all working correctly on an Exadata V2. It looks like when developing the Exadata plugin for Enterprise manager 12c they clearly developed it on a X2 only, getting a V2 as a target into Enterprise Manager does not work out of the box. In order to get Enterprise Manager 12c to discover your Exadata V2 you need to do some extra steps.

Exadata discovery is done using the first compute node in your Exadata rack (e.g. dm01db01). The agent uses a file called databasemachine.xml which is located in your One Command directory:

[oracle@dm01db01 [+ASM1] ~]$ ls -la /opt/oracle.SupportTools/onecommand/database*
-rw-r--r-- 1 root root 15790 May 10 22:07 /opt/oracle.SupportTools/onecommand/databasemachine.xml
[oracle@dm01db01 [+ASM1] ~]$

This file is being generated with dbm_configurator.xls in the One Command directory, unfortunately for V2 owners, early One Command versions did not generate these files so you have generated it yourself. Obviously you need Excel and a Windows pc to use dbm_configurator.xls as it uses VBA (Visual Basic for Applications) to generate the One Command files.

  • On the first node in the rack scp the following 2 files from /opt/oracle.SupportTools/onecommand:
    1. config.dat
    2. onecommand.params
  • Download OneCommand: Patch 13612149
  • Unzip the file windows host
  • Extract the tarbal onecmd.tar
  • Open dbm_configurator.xls in Excel
  • Enable macro’s withing excel
  • Click on the import button in the top left and locate the onecommand.params file (make sure that config.dat is in the same directory)
  • Check if the imported data is still correct
  • Click the generate button
  • Click the create config files button

After this upload at least the databasemachine.xml to /opt/oracle.SupportTools/onecommand on your first node in your rack.

Next step is to correct the Infiniband naming of the compute node HBA’s, right now on a V2 these are as follow:

[root@dm01db01 mlx4_0]# ibnodes | grep dm01db
Ca     : 0x00212800013f1242 ports 2 "dm01db04 HCA-1"
Ca     : 0x00212800013f12da ports 2 "dm01db02 HCA-1"
Ca     : 0x00212800013f111e ports 2 "dm01db03 HCA-1"
Ca     : 0x00212800013f2672 ports 2 "dm01db01 HCA-1"

Unfortunately the agent discovery process is looking for a naming convention that goes ‘hostname S ip-address HCA-1′. Fortunately Oracle provided us with a script to correct this: /opt/oracle.cellos/ When you run this script on a V2 not much will happen, it is broken on a V2 system. The problem is in the infiniband bond naming:

[root@dm01db01 ~]# grep IFCFG_BONDIB /opt/oracle.cellos/
  local IFCFG_BONDIB=/etc/sysconfig/network-scripts/ifcfg-bondib
        local addr=`awk -F= 'BEGIN {IGNORECASE=1} /^IPADDR=[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$/ {print $2}' $IFCFG_BONDIB$id 2>/dev/null`
[root@dm01db01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-bondib
cat: /etc/sysconfig/network-scripts/ifcfg-bondib: No such file or directory
[root@dm01db01 ~]# 

So Exadata V2 IB bond has a different, it is actually called bond0 instead of bondib:

[root@dm01db01 ~]# ifconfig bond0
bond0     Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  
          inet addr:  Bcast:  Mask:
          inet6 addr: fe80::221:2800:13f:2673/64 Scope:Link
          RX packets:55048256 errors:0 dropped:0 overruns:0 frame:0
          TX packets:56638365 errors:0 dropped:21 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:12207158878 (11.3 GiB)  TX bytes:18646886557 (17.3 GiB)

[root@dm01db01 ~]# 

So instead of using the broken script, fix it manually:

[root@dm01db01 ~]# dcli -g dbs_group -l root "echo -n `hostname -s` S `ifconfig bond0 | grep 'inet addr' | cut -f2 -d: | cut -f1 -d' '` HCA-1 > /sys/class/infiniband/mlx4_0/node_desc"

If all went well you should end up with the following:

[root@dm01db01 ~]# ibnodes | grep dm01db
Ca     : 0x00212800013f1242 ports 2 "dm01db01 S HCA-1"
Ca     : 0x00212800013f12da ports 2 "dm01db01 S HCA-1"
Ca     : 0x00212800013f111e ports 2 "dm01db01 S HCA-1"
Ca     : 0x00212800013f2672 ports 2 "dm01db01 S HCA-1"

After these changes the guided discovery of your Exadata should now run as is described in the cloud control manual.