[ale] Anyone doing Oracle RAC on Linux?

Mon Oct 15 13:58:56 EDT 2007

Thanks.

Apparently you missed that I did setup raw devices using udev.

So far I do have Clusterware up and running using OCR and voting disk on
those raw devices.  The raw devices are bound to multipath devices for
our dual HBA systems.  The multipathing used was EMC's PowerPath.  It
makes Pseudo-devices named emcpowera, emcpowerb, emcpowerc etc...

The EMC Clariion has two SPs for redundancy so we have two SAN paths
from the array to the switch.  Since we have two HBAs (QLE2460) in the
servers we have two SAN paths from each host to the switch.  This means
each host can see every LUN 4 times (2 x 2). 

So for example my first LUN from the array is seen as /dev/sdc,
/dev/sdf, /dev/sdi & /dev/sdl - /dev/emcpowera is the pseudo-device that
uses all 4 of those "real" devices.  So long as any of the 4 real device
paths is accessible and I've used the pseudo-device it means my path to
storage is still valid.

So when I did my udev rules for ocr and voting disk I setup:

# ORACLE CUSTOMIZATIONS jlightner 11-Oct-2007
# Bindings for Oracle 10gR2 Clusterware/RAC/ASM/DB
# raw1 = OCR, raw2 = Voting Disk, raw3 & raw4 = Disks for ASM
   ACTION=="add", KERNEL=="emcpowera1", RUN+="/bin/raw /dev/raw/raw1 %N"
   ACTION=="add", KERNEL=="emcpowera2", RUN+="/bin/raw /dev/raw/raw2 %N"
   ACTION=="add", KERNEL=="emcpowerb1", RUN+="/bin/raw /dev/raw/raw3 %N"
   ACTION=="add", KERNEL=="emcpowerc1", RUN+="/bin/raw /dev/raw/raw4 %N"
# Permissions for Oracle 10gR2 Clusterware/RAC/ASM/DB
# Note no redundant OCR and Voting Disks here - Using external
redundancy
# OCR  (root is default owner so not set here - must be root)
   KERNEL=="raw[1]*", GROUP="oinstall", MODE="640"
# Voting Disk
   KERNEL=="raw[2]*", OWNER="oracle", GROUP="oinstall", MODE="640"
# ASM (DB)
   KERNEL=="raw[3-5]*", OWNER="mngpora", GROUP="dba", MODE="660"
# END OF ORACLE CUSTOMIZATIONS

As can be seen I've used the pseudo-device rather than any of the "real"
devices so that the redundancy works if any of the "real" device paths
fail.

Since I successfully installed Clusterware (10gR2 10.2.0.1) using the
OCR and voting disk describe above it appears that at least works with
the multipathing.   However, I've not yet done the ASM/DB install so
will have to let you know.   

As noted previously I already know ASMLIB doesn't work with the
pseudo-devices but ASMLIB is there so you can try to avoid the need for
raw devices - basically just saves a step - instead of binding block to
raw then adding to ASM disk group its purpose is to allow you to add the
block devices to the ASM disk group directly.

Once we've done the ASM on raw and the DB install (or failed at it) I'll
post an update.

-----Original Message-----
From: ale-bounces at ale.org [mailto:ale-bounces at ale.org] On Behalf Of
Philip Rodgers
Sent: Monday, October 15, 2007 1:36 PM
To: Atlanta Linux Enthusiasts
Subject: Re: [ale] Anyone doing Oracle RAC on Linux?

no the rdac drivers would not compile.  I got the raw devices to work
you just have to use the udev rules file instead of the raw devices
file.  Also you can't start them with /etc/init.d/rawdevices as that
does not exist any longer.

The rdac drivers are only for redundancy.  It worked just fine when we
had single connection to the array so the raw devices will be fine.

If you find a way to get dual hbas to work with Enterprise 5 i would
like to know how you did it.  We just went back to RHEL 4 and
everything was great.

On 10/14/07, Jeff Lightner <jlightner at water.com> wrote:
> Which drivers wouldn't compile for raw?
>
> The raw stuff is still there in RHEL 5 (albeit deprecated).  You have
to
> use udev to configure raw devices though because
> /etc/sysconfig/rawdevices doesn't exist after RHEL 4.  I didn't have
to
> compile anything to add this support.
>
> We've gotten the Clusterware (Oracle 10gR2 10.2.0.1) to install using
> raw devices.
>
> We'll start on ASM (not ASMLIB) and DB install tomorrow using the raw
> devices I configured for that.  Is it relinking for this that you're
> saying didn't work?
>
> It is much like pulling teeth though.  10gR2 is "supported" on RHEL 5
> but not "documented" very well.  Lots of research at every step that
has
> an issue but hey that's how we learn right?
>
> P.S.  In case it helps anyone - Most documentation I've run across
> equates RHEL 5 to OEL 5 and SLES 10.
>
> -----Original Message-----
> From: ale-bounces at ale.org [mailto:ale-bounces at ale.org] On Behalf Of
> Philip Rodgers
> Sent: Saturday, October 13, 2007 3:54 PM
> To: Atlanta Linux Enthusiasts
> Subject: Re: [ale] Anyone doing Oracle RAC on Linux?
>
> I couldn't get the drivers to compile.  Apparently the SCSI code in
> the kernel changed at some point and all the include file weren't in
> the kernel source.  It worked the HA worked fine as long as i didn't
> worry about redundancy.
>
> On 10/10/07, Jeff Lightner <jlightner at water.com> wrote:
> > Phillip,
> >
> > Can you elaborate on what issue you saw using redundant paths for
raw
> > devices on RHEL5?
> >
> > Information gleaned in my install to date:
> > Well it appears ASMLIB doesn't work with EMC PowerPath as of version
> 4.4
> > of the latter.  The release notes for 4.3 talk about how to use it
and
> > the ones in 4.4 say not to use it at all (and remove it).   Since
> we're
> > doing RHEL5 we have to use 5.01 and versions after 4.4 don't mention
> > ASMLIB so my guess is they removed support from 4.4 and all
subsequent
> > versions.
> >
> > My coworker did an strace and found it had to do with the way the
disk
> > (LUN) responds to one call.  Apparently the multipath pseudo-devices
> > (LUNs - called emcpower[A-Z] in PowerPath) don't respond the way a
> > regular disk would so ASMLIB complains that it isn't a "partition".
> He
> > found comments that this was the issue for RDAC as well.
> >
> > I was going to fall back and use "raw" with udev (had already worked
> out
> > udev raw device entries for ocr and votingdisk) but saw your comment
> > about issues with redundant fibre paths for raw devices.   This
wasn't
> > an issue for ocr/voting disks on RHEL3.
> >
> >
> > P.S.  To anyone else who wants to do RAC on RHEL5 I'd recommend
going
> > with OCFS2 out of the gate.  It supports OCR, Voting Disks and the
> > Database storage.  Apparently raw devices are deprecated as of RHEL5
> so
> > may not be available in future.  I'm just proceeding down this path
> > because we wanted to play with ASM rather than OCFS2.
> >
> > -----Original Message-----
> > From: ale-bounces at ale.org [mailto:ale-bounces at ale.org] On Behalf Of
> > Philip Rodgers
> > Sent: Thursday, October 04, 2007 8:41 PM
> > To: Atlanta Linux Enthusiasts
> > Subject: Re: [ale] Anyone doing Oracle RAC on Linux?
> >
> > We have tested Oracle RAC on RHEL 4 and RHEL 5 with ASM (no OCFS)
We
> > decided to go with RHEL 4 because I never could get the RDAC drivers
> > for the fiber HBA to work with RHEL 5.  It was a StorageTek 2450 by
> > the way.  It was very straight forward install on RHEL 4.  There is
an
> > Metalink article concerning install on RHEL 5.  Also the handling of
> > raw devices is different on RHEL 5 which added to the install
> > difficulties but it worked great as long as I didn't use redundant
> > fiber connections.  Perhaps someone could shed some light on what to
> > do when using RDAC linux on RHEL 5.
> >
> > On 10/4/07, Jeff Lightner <jlightner at water.com> wrote:
> > >
> > >
> > >
> > > We currently have Oracle RAC on RHEL3 using OCFS for the
datafiles.
> > There
> > > is a project underway to do RAC on RHEL5 and we're trying to
decide
> > between
> > > using ASM and OCFS2.
> > >
> > > Is anyone willing to share what they decided if they already did
> this
> > on a
> > > 2.6 kernel based Linux and why they decided it?
> > > ----------------------------------
> > > CONFIDENTIALITY NOTICE: This e-mail may contain privileged or
> > confidential
> > > information and is for the sole use of the intended recipient(s).
If
> > you are
> > > not the intended recipient, any disclosure, copying, distribution,
> or
> > use of
> > > the contents of this information is prohibited and may be
unlawful.
> If
> > you
> > > have received this electronic transmission in error, please reply
> > > immediately to the sender that you have received the message in
> error,
> > and
> > > delete it. Thank you.
> > > ----------------------------------
> > >
> > > _______________________________________________
> > > Ale mailing list
> > > Ale at ale.org
> > > http://www.ale.org/mailman/listinfo/ale
> > >
> >
> >
> > --
> > Philip Rodgers
> > philipfromga at gmail.com
> > "We are spending ourselves into oblivion because we can't tax
> > ourselves into prosperity"  Herman Cain
> > _______________________________________________
> > Ale mailing list
> > Ale at ale.org
> > http://www.ale.org/mailman/listinfo/ale
> > ----------------------------------
> > CONFIDENTIALITY NOTICE: This e-mail may contain privileged or
> confidential information and is for the sole use of the intended
> recipient(s). If you are not the intended recipient, any disclosure,
> copying, distribution, or use of the contents of this information is
> prohibited and may be unlawful. If you have received this electronic
> transmission in error, please reply immediately to the sender that you
> have received the message in error, and delete it. Thank you.
> > ----------------------------------
> > _______________________________________________
> > Ale mailing list
> > Ale at ale.org
> > http://www.ale.org/mailman/listinfo/ale
> >
>
>
> --
> Philip Rodgers
> philipfromga at gmail.com
> "We are spending ourselves into oblivion because we can't tax
> ourselves into prosperity"  Herman Cain
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://www.ale.org/mailman/listinfo/ale
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://www.ale.org/mailman/listinfo/ale
>

-- 
Philip Rodgers
philipfromga at gmail.com
"We are spending ourselves into oblivion because we can't tax
ourselves into prosperity"  Herman Cain
_______________________________________________
Ale mailing list
Ale at ale.org
http://www.ale.org/mailman/listinfo/ale