ODA: starting Guest VM, got Domain 'VM_NAME' does not exist

After we applied the patch 12.1.2.11.0 to our ODA X-5, we can't start our guest VM:

$ oakcli start vm vm4

OAKERR:7007 Error encountered while starting VM - Error: Domain '<VM_NAME>' does not exist.

Domain, guest and virtual machine are often used interchangeably, but there are subtle differences. A domain is a configurable set of resources, including memory, virtual CPUs, network devices and disk devices, in which virtual machines run. So first, let's take a look of the vm.cfg:

$ oakcli show vm vm4

The Resource is : vm4

        AutoStart       :       restore

        CPUPriority     :       100

        Disks           :       |file:/OVS/Repositories/apexrepo/.A

                                CFS/snaps/vm4/VirtualMach

                                ines/vm4/1e5f4bcd6d6948f0

                                a177595b260de280.img,xvda,w|

        Domain          :       XEN_PVM

        DriverDomain    :       False

        ExpectedState   :       offline

        FailOver        :       false

        IsSharedRepo    :       true

        Keyboard        :       en-us

        MaxMemory       :       2048M

        MaxVcpu         :       2

        Memory          :       2048M

        Mouse           :       OS_DEFAULT

        Name            :       vm4

        Networks        :

        NodeNumStart    :       0

        OS              :       OL_5

        PrefNodeNum     :       0

        PrivateIP       :       None

        ProcessorCap    :       100

        RepoName        :       apexrepo

        State           :       Offline

        TemplateName    :       otml_OL6U7_PVHVM

        VDisks          :       |oakvdk_vm4_vdisk0_apexre

                                po||oakvdk_vm4_vswap_apex

                                repo|

        Vcpu            :       2

        cpupool         :       default-unpinned-pool

        vncport         :       None

ok, easy fix, right?

find out the vlan:

$ oakcli show vlan

reconfigure the interface:

$ oakcli configure vm vm4 -network "['bridge=vlan914b1']"

also, check your vm.cfg, if you have extra=.... line, remove it:

cat vm.cfg:

vif = ['']
name = 'vm4'
extra = 'NODENAME=vm4' >>>>>>>>>> This needs to be deleted and then vm can be started on 12.1.2.11
builder = 'hvm'
cpus = '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23'
vcpus = 2
memory = 2048
cpu_cap = 0
vnc = 1
serial = 'pty'
disk =
[u'file:/OVS/Repositories/odarepo1/VirtualMachines/vm4/68c32afe2ba8493e89f018a970c644ea.img,xvda,w']
maxvcpus = 2
maxmem = 2048

now, try again:

$ oakcli start vm vm4

Started VM : vm4 on Node Number : 0

Reference:  Doc ID 2285306.1

Easy OFSAA DR testing / DR solution

1. All the ODA VMs have DNS entries. So, for DR testing or DR switch over, we just need change the DNS entries/alias to point to the different IPs. e.g.:
    OFSAA app server (i.e.: ofsaaprd): 10.120.132.117
    OFSAA Weblogic server (i.e.: ofsawlsprd) and Web Server (i.e.: ofsawebprd): 10.120.132.118
    Database server (i.e.: ofsadbprd): oda-scan or 10.120.132.111 (ODA base)

2. Get a copy of the OFSAA files (backup every day, maybe)

tar -pczf /cloudfs/backup/ofsaa_tar/ftpshare.tar.gz -C / u03/app/ftpshare

The following script will backup all the OFSAA apps, for DR testing, just need ftpshare copied over to the DR site.

# cat tarOFSAA4DR.sh

#!/bin/bash

# tarOFSAA4DR.sh

#-------------------------------------------------------------------------------

# Description:

#   tar OFSAA app for DR

# Example:

#

# Modification History

#   07/17/2017 xxxxxxx Created

#

#-------------------------------------------------------------------------------

# User defined parameters

#

nowDate=$(date '+%Y%m%d')

hostName=$(hostname)

tarDir=/mfs/backup/ofsaa_tar

logFile=/mfs/logs/OFSAA/GuestVMBackup/tarOFSAA4DR_${nowDate}.log

ftpshareFile=${hostName}_ftpshare_${nowDate}.tar.gz

ofsaaFile=${hostName}_ofsaa_${nowDate}.tar.gz

odiFile=${hostName}_odi_${nowDate}.tar.gz

pidLst=""

echo "################################################################################" | tee -a ${logFile}

echo "# BEGIN TAR GZIP OFSAA,FTPSHARE,ODI $(date '+%Y-%m-%d %H:%M:%S')" | tee -a ${logFile}

echo "################################################################################" | tee -a ${logFile}

echo "$(date '+%Y-%m-%d %H:%M:%S') Tar Directory.......: "${tarDir} | tee -a ${logFile}

echo "$(date '+%Y-%m-%d %H:%M:%S') FTPSHARE FileName...: "${ftpshareFile} | tee -a ${logFile}

echo "$(date '+%Y-%m-%d %H:%M:%S') OFSAA FileName......: "${ofsaaFile} | tee -a ${logFile}

echo "$(date '+%Y-%m-%d %H:%M:%S') ODI FileName........: "${odiFile} | tee -a ${logFile}

echo "$(date '+%Y-%m-%d %H:%M:%S') Script Log FileName.: "${logFile} | tee -a ${logFile}

echo "" | tee -a ${logFile}

echo "$(date '+%Y-%m-%d %H:%M:%S') EXECUTING: tar ${tarDir}/${ofsaaFile}" | tee -a ${logFile}

tar -pczf ${tarDir}/${ofsaaFile} -C / u03/app/ofsaa &

aPid=$!

if [ "${aPid}" = "0" ]; then

 echo "$(date '+%Y-%m-%d %H:%M:%S') ERROR: HALTING!!!!"

 exit 1

else

 pidLst="$pidLst ${aPid}"

fi

echo "$(date '+%Y-%m-%d %H:%M:%S') EXECUTING: tar ${tarDir}/${ftpshareFile}" | tee -a ${logFile}

tar -pczf ${tarDir}/${ftpshareFile} -C / u03/app/ftpshare &

aPid=$!

echo ${aPid}

if [ "${aPid}" = "0" ]; then

 echo "$(date '+%Y-%m-%d %H:%M:%S') ERROR: HALTING!!!!"

 exit 1

else

 pidLst="$pidLst ${aPid}"

fi

#echo "$(date '+%Y-%m-%d %H:%M:%S') EXECUTING: tar ${tarDir}/${odiFile}" | tee -a ${logFile}

#tar -pczf ${tarDir}/${odiFile} -C / u03/app/odi &

#aPid=$!

if [ "${aPid}" = "0" ]; then

 echo "$(date '+%Y-%m-%d %H:%M:%S') ERROR: HALTING!!!!"

 exit 1

else

 pidLst="$pidLst ${aPid}"

fi

echo "tar waiting list: =$pidLst=..." | tee -a $logFile

wait $pidLst

sleep 60

echo "All tars are done. Time at `date`."  | tee -a $logFile

echo "$(date '+%Y-%m-%d %H:%M:%S') LIST FILES..." | tee -a ${logFile}

ls -al ${tarDir}/*${nowDate}*.tar.gz | tee -a ${logFile}

echo "" | tee -a ${logFile}

echo "$(date '+%Y-%m-%d %H:%M:%S') COMPLETE" | tee -a ${logFile}

echo "$(date '+%Y-%m-%d %H:%M:%S') Tar Directory.......: "${tarDir} | tee -a ${logFile}

echo "################################################################################" | tee -a ${logFile}

echo "# END TAR GZIP OFSAA,FTPSHARE,ODI $(date '+%Y-%m-%d %H:%M:%S')" | tee -a ${logFile}

echo "################################################################################" | tee -a ${logFile}

exit 0

3. Copy the ftpshare to the DR site and untar it.

tar -pzxvf /cloudfs/backup/ofsaa_tar/btcodaasp003v_ftpshare.tar.gz -C /

4. Prepare the database
    1). For DR testing, you convert the physical standby to snapshot standby and after done the DR testing, switch back to physical standby. If your master database and standby have the same dbname, add a unique name to the service_names:
     From DR database – sqlplus - add a new service name ofsdbprod:
alter system set service_names = 'ofsdbroc, ofsdbprod' scope=memory;

DGMGRL> convert database ofsaadb to snapshot standby;

DGMGRL> convert database ofsaadb to physical standby;

    2). For DR switchover, you switch the database too.

That's it. so simple and easy!

Access ODA ILOM

1, on ODA BASE, run oakcli to get the ip address for ILOM:

# oakcli show server

 ILOM Address             : 10.60.16.212

2a. access web interface
 
  http://10.60.16.212

2b. access ILOM CLI
 
  ssh -l root 10.60.16.212

login/connect as root, enter password.

Automatic Backup of VMs on ODA X5-2

I was looking for the best practice to backup ODA X5-2 guest VMs and found this post on OTN. Got lots of helpful info, but did not find an automatic way to backup the VMs. It's an old post, so I post my solution here.  Credits to many people replied to the above post .


Ref: ODAVP: HowTo backup/restore your VM Guest (Doc ID 1633166.1)


# cat backupGuestVM.sh


#!/bin/bash
# backupGuestVM.sh
# Description: Used to backup guest VM and it's vDisk.
#
# Modification History
#   10/13/2016 jx  Created
#
#-------------------------------------------------------------------------------
if [ ! $# -gt 0 ]; then
    echo "Usage: $0 all"
    echo "   or: $0 [gVM1 | gVM1 gVM2 ...]"
    exit;
fi
# Variables and functions
lpCnt=0
ttlLp=100
secWait=20
arDir=/cloudfs/backup
#arDir=/u02/app/oracle/oradata/datfinmprod
srcDir=u01/app/sharedrepo
logFile=/tmp/backupGuestVM.log
execOakCmd() {
    local oakAct=$1
    local vmList=$2
    local pidList=""
    echo "local cmd: =$oakAct=, vmList: =$vmList=..." | tee -a $logFile
    for aVM in $vmList; do
        cmd="oakcli $oakAct vm $aVM"
        echo "cmd is: =$cmd=..." | tee -a $logFile
        $cmd &
        pidList="$pidList $!"
    done
    echo "oak cmd $oakAct waiting list: =$pidList=..." | tee -a $logFile
    wait $pidList
}
#------------------------------------------------------------------------------
# Create a timestamp in a logfile
echo "backup guest VMs started at `date`..." | tee $logFile
# Create backup directory
todayStr=`date +'%Y%m%d'`
[ ! -d "$arDir/$todayStr" ] && ( mkdir $arDir/$todayStr || exit )
inVMs=$*
echo "passed in these VMs: =$inVMs=..." | tee -a $logFile
# Check if the passed in VMs are valid resources
vmLst=$inVMs
allVMLst=$( oakcli show vm|cut -f 2 -|grep -v NAME|tr -s '\n' ' ' | xargs )
if [ "$inVMs" = "all" ]; then
    vmLst=$allVMLst
else
    for aVM in $inVMs; do
        echo "$allVMLst" | grep -i $aVM
        if [ ! $? -eq 0 ]; then
            vmLst=${vmLst/$aVM/}
        fi
    done
    vmLst=$( echo $vmLst | awk '{$1=$1};1' )
fi
echo "valid VM list =$vmLst=..." | tee -a $logFile

# vm list is not null or empty, continue....
#------------------------------------------------------------------------------
if [ ! -z "$vmLst" -a "$vmLst" != " " ]; then
    # stop all the guest VMs
    echo "stop all the vm start at `date`..." | tee -a $logFile
    execOakCmd "stop" "$vmLst"
    # make sure all the VMs are offline
    vmArr=( $vmLst )
    vmCnt=${#vmArr[@]}
    echo "vmCnt: =$vmCnt=..." | tee -a $logFile
    # to prevent infinite loop, loop only 100 times
    while [[ $vmCnt -gt 0 && $lpCnt -lt $ttlLp ]]; do
        lpCnt=$[$lpCnt+1]
        for aVM in $vmLst; do
            isOffln=$( oakcli show vm | grep $aVM | grep OFFLINE )
            vmStatus=$( echo $? )
            if [ "$vmStatus" = "1" ]; then
                echo "Waiting $secWait seconds for =$aVM= vm goes offline ..." | tee -a $logFile
                sleep $secWait
            else
                vmCnt=$[$vmCnt-1]
            fi
        done
        echo "Waiting loop, current count: =$lpCnt=..." | tee -a $logFile
    done
    echo "All VMs are now offline at `date`...lpCnt: =$lpCnt=, vmCnt: =$vmCnt=..." |tee -a $logFile
# All VMs in the list are offline, do backup now....
#-------------------------------------------------------------------------------
    pidLst=""
    echo "create archives start at `date`..." |tee -a $logFile
    for aVM in $vmLst; do
        vmRepo=$( oakcli show vm | grep -i $aVM | cut -f 7 | awk '{$1=$1};1' )
        echo "vm: =$aVM=, repo: =$vmRepo=..." | tee -a $logFile
        dirLst=$( dir -d /$srcDir/$vmRepo/.ACFS/snaps/*$aVM* )
        for aDir in $dirLst; do
            # tar file name
            arName=${aDir##*"/"}
            arName=${arName#"oakvdk_"}
            arFile="$arDir/$todayStr/$arName.tar.gz"
            # source directory
            inFile=${aDir#"/"}
            tarCmd="tar -czf $arFile -C / $inFile"
            echo "Starting tar command =$tarCmd=... " | tee -a $logFile
            $tarCmd &
            pidLst="$pidLst $!"
        done; # loop thru the directories this vm has, which could be vm itself and vdisks;
    done; # loop thru the vmList
    echo "tar waiting list: =$pidLst=..." | tee -a $logFile
    wait $pidLst
    echo "All backups are done. Time at `date`."  | tee -a $logFile
    # After all backup done, start the VMs again
    execOakCmd "start" "$vmLst"
fi; # vmLst is not null or space
echo "backup succeeded at `date`." | tee -a $logFile

Find the ODA Memory Allocation

To find out the ODA BASE memory allocation, on DOM0, you can use oakcli show oda_base command:

# oakcli show oda_base
ODA base domain
ODA base CPU cores      :8
ODA base domain memory  :64
ODA base template       :/OVS/oda_base_12.1.2.7.gz
ODA base vlans          :['net1', 'net2', 'vlan321', 'vbr1']
ODA base current status :Running

When deploy the ODA, you decide how much memory to allocate to the ODA base and the rest will be used for VMs or set aside for future growth. In the above example, it was allocated 64G for the ODA_BASE.

Use xm info to find out the available memories on the ODA.

# xm info
xm info
host                   :
release                : 2.6.39-400.276.1.el5uek
version                : #1 SMP Wed Jan 27 15:31:57 PST 2016
machine                : x86_64
nr_cpus                : 72
nr_nodes               : 2
cores_per_socket       : 18
threads_per_core       : 2
cpu_mhz                : 2294
hw_caps                : bfebfbff:2c100800:00000000:00007f40:73fefbff:00000000:00000021:000037ab
virt_caps              : hvm hvm_directio
total_memory           : 262010                                // Total physical installed.
free_memory            : 188123                               // Available for VMs, etc.
free_cpus              : 0
xen_major              : 4
xen_minor              : 1
xen_extra              : .3OVM
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
xen_commandline        : dom0_mem=4096M crashkernel=256M@64M extra_guest_irqs=64,2048 nr_irqs=2048
cc_compiler            : gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)
cc_compile_by          : mockbuild
cc_compile_domain      : us.oracle.com
cc_compile_date        : Wed Sep 23 23:17:43 PDT 2015
xend_config_format     : 4