[ale] School Project to Create Distributed Filesystem

[Greg Freemyer]

I think you have it.

As to how the master is assigned, it depends on how you config heartbeat.

You can weight all the servers equally, or you can put a preference on
one (or more.  I think the weighting for selecting the master is
flexible.)

Greg

On Wed, Feb 25, 2009 at 4:20 PM, Omar Chanouha <ofosho at gatech.edu> wrote:
> Okay, so I would have a cluster of servers that all have iSCSI
> servers. I use linux-ha to dedicate a master server. The master then
> mounts all the iSCSI drives via an mdraid, then shares the drive via
> nfs. When a client connects, it gets routed to the Master over nfs and
> gets all its files from it. The heartbeat is also in charge of
> maintaining an up-to-date mdraid array, and reasigning a master if
> neccesary.
>
> Is that correct? If so, it seems as though there is now a funnel at
> the master. The master now has to accept and serve all incoming
> connections. With this algorithm, we could not have two or more
> different servers accept incoming connections at the same time?
>
> Just trying to be clear before I present this to my professor.
>
> -OFosho, Miami Dolphin and Open Source Aficionado
>
> On Wed, Feb 25, 2009 at 3:56 PM, Greg Freemyer <greg.freemyer at gmail.com> wrote:
>> Very close, but you use heartbeat / linux-ha to run (mdraid, the
>> filesystem and nfx export) on only one server at a time.  Otherwise
>> you could have data consistency issues between the nodes.  That is why
>> true cluster / distributed filesystems are a killer to write.
>>
>> heartbeat is effectively a cluster controller.
>>
>> For instance, you could configure it to know about 8 nodes and that it
>> needs at least 6 of the nodes available for there to an operational
>> cluster.  (Raid -6 is operation even if 2 nodes have failed.)
>>
>> Then you have a 9th IP that is called the cluster IP that floats
>> between the nodes.  All of the clients connect exclusively to the
>> cluster IP.
>>
>> Once the cluster is formed (6 or more nodes), heartbeat controls which
>> of the nodes is the master.
>>
>> On the master it will use IP aliasing to respond to any and all tcp/ip
>> connections to the cluster IP.  Further heartbeat you will need to
>> configure to launch mdraid, form the raid array, mount the filesystem,
>> and export it via nfs.
>>
>> If you are familiar with init scripts (/etc/init/*) then you are
>> familiar with the scripts heartbeat uses.   Heartbeat is actually a
>> superset of those scripts in that heartbeat scripts also support a
>> "monitor" command.  So you just have heartbeat invoke:
>>
>> rcmdraid start
>> rcfilesystem start
>> rcnfs start
>>
>> on the master, and on all the slaves it calls the above with stop.
>>
>> Fortunately the vast majority of this already exists, but I'm not
>> aware of anyone putting all the pieces together in this way.
>>
>> FYI: Heartbeat also provides a reliable comm channel if you need it.
>> I don't think you will, but you might.
>>
>> Greg
>>
>> On Wed, Feb 25, 2009 at 3:09 PM, Omar Chanouha <ofosho at gatech.edu> wrote:
>>> Greg/All, Let me get one more shot at the belt.
>>>
>>> Have a cluster of servers each sharing their files via iSCSI. Each
>>> server then keeps an array of all the other servers via mdraid.
>>> Linux-HA is used on the servers to maintain an up-to-date array of
>>> iSCSI drives. Each server shares the combined RAID via an nfs server.
>>> When a client wants to connect, all it needs to do is connect to one
>>> of the servers over nfs and it will have all the files.
>>>
>>> As you said, the robustness comes into play with the mdraid update
>>> scripts/making nfs work properly.
>>>
>>> Is that right?
>>>
>>> I will have to ask the teacher if he will allow that. He may not.
>>>
>>> Thanks,
>>>
>>> -OFosho, Miami Dolphin and Open Source Aficionado
>>>
>>> On Wed, Feb 25, 2009 at 7:26 AM, Jim Kinney <jim.kinney at gmail.com> wrote:
>>>> 2009/2/24 Ken Ratliff <forsaken at targaryen.us>:
>>>>>
>>>>> On Feb 24, 2009, at 8:25 PM, Brian Pitts wrote:
>>>>>
>>>>> Another one worth checking out is gluster. It uses FUSE.
>>>>>
>>>>> http://www.gluster.org/
>>>>>
>>>>>
>>>>> Gluster is.... finicky. We have it deployed in a few production clusters and
>>>>> it works well when it's working, but when it breaks, it tends to break hard.
>>>>
>>>> Sounds like a good project for students to work on :-) It needs help.
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Ale mailing list
>>>>> Ale at ale.org
>>>>> http://mail.ale.org/mailman/listinfo/ale
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> --
>>>> James P. Kinney III
>>>> _______________________________________________
>>>> Ale mailing list
>>>> Ale at ale.org
>>>> http://mail.ale.org/mailman/listinfo/ale
>>>>
>>> _______________________________________________
>>> Ale mailing list
>>> Ale at ale.org
>>> http://mail.ale.org/mailman/listinfo/ale
>>>
>>
>>
>>
>> --
>> Greg Freemyer
>> Litigation Triage Solutions Specialist
>> http://www.linkedin.com/in/gregfreemyer
>> First 99 Days Litigation White Paper -
>> http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf
>>
>> The Norcross Group
>> The Intersection of Evidence & Technology
>> http://www.norcrossgroup.com
>> _______________________________________________
>> Ale mailing list
>> Ale at ale.org
>> http://mail.ale.org/mailman/listinfo/ale
>>
>
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
>



-- 
Greg Freemyer
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
First 99 Days Litigation White Paper -
http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf

The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com