This is really going the last post of my series on Oracle VM Server / VM Manager on inexpensive hardware.
Last week a second Dell Power Edge arrived, followed by a little NAS/iSCSI System, the ES-2100 from Eurostor, which is rebranded Thecus N5200 Pro. I do link Eurostor because i made some very nice contact with their tech support.
After running the Oracle VM Server on a 2.4 GHz Core 2 Duo Xeon with 4 Gb Ram for about 70 days non-stop, we decided to do the next step: Incarnating a second server with a shared storage.
The one server runs an paravirtualized OEL5 with 2 GB Ram which itself runs an Oracle 11g test instance with medium load, a hardware virtualized Windows XP with 512 MB Ram that runs a Jetty with a few services and since 2 weeks a hvm Debian that serves as a mailrelay for that Exchange of ours… Which has a now really less load as SpamAssasin takes care of all.
Setting up the second Dell was flawless, nothing new.
The iSCSI was another thing… First i deleted the RAID6 as we decided to go for RAID5. Stupid me set disk usage to 100%, went for the weekend, came back on monday and saw: Wow, no space for the iSCSI target. Damn it, all timeplans went bazoo… So deleting the RAID once again and back to start, this time with 20% for Disk Usage (you never know) and 80% for one iSCSI target (if this was my machine, i really had a purpose for 1.5TB storage… but here.. *sigh*).
So, another 8 hours later, i bought a cheap 8 port Gigabit switch, set up the ES-2100 for link aggregation and connected it to both Oracle VM Servers.
I roughly followed the steps described here, but as i changed some steps, let me describe them:
rpm -Uvh iscsi-initiator-utils-6.2.0.742-0.5.el5.i386.rpm
iscsiadm -m discovery -t sendtargets -p 139.185.48.249
Example of removing a node:
iscsiadm -m node -p 10.2.0.250:3260,3 -T iqn.1992-04.com.emc:cx.apm00070202838.b0 -o delete
Listing the remaining:
iscsiadm -m nodeand having a new partition under /proc/partitions after
service iscsi restart
The ocfs2 cluster configuration is as simple as described in the linked Oracle document. I recommend adding names and ipaddresses corresponding to the one in /etc /ocfs2/cluster.conf to /etc /hosts, as the o2cb services won’t start otherwise. One thing Oracle forgot to mention is to open port 7777 on both machines in the iptables configuration.
At first i made the mistake to mkfs.ocfs2 the device and forgot to create a partition. This worked for whatever reason, but i destroyed the filesystem and created a partition with fdisk (new partition, primary, the whole thing).
Next, i didn’t follow Oracle but decided the following:
/dev/sdb1 /OVS ocfs2 defaults 1 0
So in the end we have a safe setup with hardware costs under 5k € and a setup time from about 6 or 7 days which brought some good knowledge and know-how. I think we wouldn’t have achivied this based on a VMWare solution brought by external consultants. Maybe that would have ended like the last Dilbert in that series 
.
To bring some variety to this blog, here’s a picture of the current setup:
I one of the google visitors or the 2 readers have any questions, feel free to ask 
As a last post in the series (1, 2, 3, 4): Installing the latest and greatest incarnation of the Oracle Database named “Oracle 11g” on Oracle Enterprise Linux 5 running as a paravirtualized machine on Oracle VM Server.
Honestly, i was afraid of installing Oracle the first time under a linux environment but i must say, the installation was pleasantly easy, following the steps described here. No surprises at all. My host has a fixed IP adress, no problem with DHCP (although a don’t quite get why this should be any problem at all).
Fun started after stopping the database.
First: SELINUX is enabled per default under OEL5. I found the following tutorial which recommends turning SELINUX off. I disagree with that.
After installation complets, the following commands are necessary to enable Oracle to load its shared objects and to some text relocation with it:
chcon -t textrel_shlib_t /opt/app/oracle/product/11.1.0/db_1/lib/libnnz11.so chcon -t textrel_shlib_t /opt/app/oracle/product/11.1.0/db_1/lib/libclntsh.so.11 chcon -t textrel_shlib_t /opt/app/oracle/product/11.1.0/db_1/lib/libnque11.so chcon -t textrel_shlib_t /opt/app/oracle/product/11.1.0/db_1/lib/libsqlplus.so
otherwise you’ll some errors like
/opt/app/oracle/product/11.1.0/db_1/bin/ocrconfig.bin: error while loading shared libraries: /opt/app/oracle/product/11.1.0/db_1/lib/libnnz11.so: cannot restore segment prot after reloc: Permission denied
(Paths may vary) when starting the database.
Speaking of starting, Oracle comes with no batteries included, i.e. a startup script is missing.
I started with the one described here and added some lines to start the enterprise manager controll, see resources below.
In the meantime, the first import should be finished. The performance of 11g seems ok, on a virtual machine with a minimum of 1GB Ram. Maybe i’ll post some more impressions in a few weeks.
Installing the latest incarnation of Oracle Enterprise Linux 5 as a paravirtualized machine in Oracle VM Server is a bit tricky.
The OEL5 is delivered as 5 iso files, cd images they are. The Oracle VM Manager is only able to install the patched paravirtualized guest system from one installation medium. Damn it, they praise their system as blazing fast under their own VM and didn’t manage to deliver the stuff as one dvd image?
After a lot of googling i found the following script. Its a CentOS script to combine their installation mediums into on, big iso file. I thought, give it a try as CentOS is based on RedHat as well is OEL5. It ate a lot of diskspace but in the end, i got a working DVD that i mounted via the loop device.
mount -o rw,loop /OVS/tmp/oel5dvd.iso /tmp/oel5/
The next step requires the installation medium available either via http, ftp or nfs. Starting nfsd and exporting the fs is done easily via:
/etc /init.d/nfs start exportfs *:/tmp/oel5/
I had all the iso and the resulting dvd on my Oracle VM Server. The default installation is rather strict with the iptable rules.
Although Oracle VM Manager will except an nfs://127.0.0.1/… path and also do the initial patching, the installation will fail afterwards.
To get NFS working, i followed these instructions to pin all needed ports and open them in iptables.
Installation itself was after that straigt forward. I deselected all prepackaged environments since the server was destined to be a database server.
I didn’t realize that this was going to be a problem.
There is no centralized repository in the RHEL or OEL world like the one in Debian or Ubuntu. Either hunt the rpms down or subscribe to the services both manufactures offers, in my case the Oracle Unbreakable Linux Network. Hello? I just wanted to install a gcc, for example.
Their is no possibility in the gui to reuse your installation medium as a repository for “YUM” or “Pirut”.
Bummer! After a lot of searching i found this, but this method didn’t work for me. Nevertheless, i’ve learnd how to configure repositories for yum. Pirut did find the installation medium but wasn’t able to install a package due to an “uropen error unknown url type: media”.
My solution was to copy all rpms from the /Server directories of the original installation disk along with “comps-rhel5-server-core.xml” from the first disc found in /Server/repodata/ to disk, running
createrepo -g comps-rhel5-server-core.xml .and finally exporting this directory as a nfs share, configuring it to yum with the following:
[oel5] name=Oracle Enterprise Linux 5 baseurl=file:///tmp/oel5_repository enabled=1 gpgcheck=0
After a “yum clean all”, pirut can add and remove packages from the “original” medium without any hassle.
I installed the following packages:
yum install binutils yum install compat-libstdc++ yum install elfutils-libelf yum install elfutils-libelf-devel yum install gcc yum install gcc-c++ yum install glibc yum install glibc-common yum install glibc-devel yum install glibc-headers yum install libaio yum install libaio-devel yum install libgcc yum install libstdc++ yum install libstdc++-devel yum install make yum install sysstat yum install unixODBC yum install unixODBC-devel
preparing the next post which is about installing Oracle 11g on Oracle Enterprise Linux 5.
As promised, i’d like to share some experience with installing the new “Oracle VM Server”.
The product isn’t as new as Oracle wants you to believe, it’s based on Xen 3.1.1 “with many, many performance enhancements done by our engineers”. We’ll see.
After unpacking our shiny new Dell (The rabat action was entitled “My First Server”, sweet… kind of), i booted of CD, left most settings on default and made my way to the non-graphical Installer. No surprises here, Oracles Xen Version runs on their own Linux distribution (“Oracle Unbreakable Linux / Enterprise Linux”) which itself is based on Red Hat Enterprise Linux (ok, tante isn’t wrong about re branding stuff and so…). It’s quickly said that the stuff is available without re branding in it’s original form but to many people it is important that these products are certified by Oracle.
Oracle VM comes in two packages, a server and a manager component, which needs to be installed on separate machines.
Oracle VM Manager runs on both Oracle Enterprise Linux 4 and 5. I set up a VMWare Server machine on my workstation for that as i didn’t want to ruin a near perfect desktop. Installation of the manager is simple with one little caveat: The script prompts for the oc4j admin password which is “oracle”. It doesn’t want to know a new one but the default.
VM Manager installs an OC4J, an Oracle XE Database and the manager app itself which is a J2EE app.
Within this thingy you define server pools and servers.
For the later you have to install one. Put in the disc, but the machine, accept a view defaults and you’re ready to go. I put two ethernet cards in our dell for accessing the server itself and the other for the virtual machines.
I had some weird problems configuring the ntp daemon, that was all.
My first experiment was to install a fully hardware virtualized Windows XP. Creating the vm, adding diskspace and so on with Oracle VM Manager is as easy as with any other tool like the ones from VMWare. One noteworthiness is that you need to import your iso files from any install media to the server. This was easily done via ftp. I loaded the machine with XP and for later, all 5 disks of Oracle Enterprise Linux 5.
I spend nearly a day to get the bridged networking with xp up and running. I learned a lot about networking under Xen (especially this page was useful) but in the end, the on and only reason why i couldn’t reach the Windows XP domU was the bastard firewall build into XP. Bummer!
XP runs reasonable fast within the hvm environment. The only thing that felt like ages was creating the virtual disk.
From within the manager application one is able to reach a VNC server running on the Oracle VM Server entitled console. You even see the bootscreens and so on from this VNC. I first saw this on the DOAG last month, it certainly didn’t work out of the box. The Oracle VM Manager application was missing the jar files with the Tight VNC applet.
Although the VNC server is reachable with any VNC client, it’s nice to start right from within the browser. To install the applet properly on the manager machine, do the following:
rpm -ivh tightvnc-java-1.3.9-1.noarch.rpm
/opt/oc4j/j2ee/home/applications/OVS/webapp1/Class
Make sure there is a file “VncViewer.jar”. Mine was there after installing the rpm.
/opt/oc4j/java/jdk1.5.0_11/bin/keytool -genkey -alias vncviewer -validity 365 /opt/oc4j/java/jdk1.5.0_11/bin/jarsigner VncViewer.jar vncviewer
The next step on the list was installing a paravirtualized Oracle Enterprise Linux. I hoped the templates at oracle.com were paravirtualized but no, they all are hvm templates. Why the heck? So this is where the fun starts and the next post takes place…
So far, i like the concept of Oracle VM Server. The software makes it easy to start with one server which acts as a virtualisation server, a server pool master and an utility server. If your needs grow, you throw some hardware in, define a new server in the manager application, add it to the pool and you’re done. From within the manager app, you can move the virtual machines around as you like. You certainly need to have for that to work some sort of shared storage available, but i don’t think that moving the /OVS tree of the first server to some kind of iSCSI or any other SAN is a big problem.
I know that Oracle builds their product upon Open Source but in this case, i think the package works great. Buying some relatively cheap server hardware, installing and configuring two products (Oracle VM Server and VM Manager) was far less expensive than a VMWare ESX3 test installation. Where Oracle fails is their Oracle Enterprise Linux, at least in some points. But as i said, in another post.
At one point a had a problem with the Manager App. Somehow my server pools went missing, i clicked around and finally found “Restore from Manager”. Hm, well… *click* and my virtual machines were gone. After searching awhile i found out that some configuration files for VMS exists under /etc /xen on the server but i guess they aren’t use. Any changes to them were ignored. All the vm starting, stopping and restarting is done from the manager app. So be careful when fiddling around with the config files on the server!