[ale] OT converting old videos, cataloging dvds, storing forever

[Ron Frazier (ALE)]

Hi Jim,

Thanks, but the whole point of the project is to get away from tape.  Away from expensive hard to find drives, alignment and tracking problems, tape tangles, obsolete data interfaces, maintenance issues, non instant random access, etc.  I even have a scsi mini dv tape drive (if i can find it) I'll sell for $100 if anyone wants it.  That's not a direction I'm interested in going in.

Sincerely,

Ron


Jim Kinney <jim.kinney at gmail.com> wrote:

>Um. Magnetic tape is cheap and very reliable. Storage silos from ebay
>can
>hold multiple terrabytes and are Linux happy. 20 year lifespan on tape
>in
>decent environment.  So a linux storage system with enough drive space
>for
>staging restores from tape for play.
>On Oct 16, 2012 12:33 PM, "Ron Frazier (ALE)" <
>atllinuxenthinfo at techstarship.com> wrote:
>
>> HI all,
>>
>> After a few hours of Googling and reading forums and such, I've got
>some
>> new data that I wanted to share.  My little video conversion project
>> seems to be hitting a snag.  The 800 hours of videos I have are in
>the
>> category of important to me, but not critical.  If I lose them, I
>won't
>> die tomorrow.  So, the snag I'm hitting is the cost in terms of
>storage
>> cost and time to convert them.  I don't have a huge budget, certainly
>> not all at once.  I wanted to share my results, and see if you guys
>> think I'm missing something.
>>
>> In terms of time, I think it will take me 10 - 15 minutes of
>tinkering
>> with equipment, software, discs, cases, labels, etc. per one hour
>show
>> to convert.  More if I do edits and cut commercials and sync the
>sound,
>> etc.  Perhaps less if I store only to HDD and don't edit.  That adds
>up
>> to 200 hours of my time, a non trivial amount.  Obviously the video
>tape
>> must play, but I don't necessarily have to watch it.
>>
>> Here is what I believe to be the state of the art in storage options.
>>
>> I'm not considering other types of tape at this point.  I'm also not
>> considering SSD, based on data retention issues I mentioned in
>another
>> post.
>> Option 1 is using HDD's to store the videos.
>> Option 2 is some other type of disc, bluray, dvd, or mdisc.
>> Option 3 is online storage.
>> Based on my reading, I have concerns about the long term reliability
>of
>> bluray, so I'm probably dismissing it.
>>
>> Let's discuss discs first.  Assume a single layer single sided DVD
>can
>> hold ~1 hour of good quality video at a data rate of about 8 Mbps. 
>For
>> a HDD, assume 4.7 GB / hr of video.  Note, in some of the forums
>where
>> people burn LOTS of DVD's and spend hours checking their error rates
>> etc, it was mentioned that 16x write speed DVD's are not reliable. 
>So,
>> if available, I'm mentioning only 8x DVD's here.  There are also many
>> types of dyes, and very confusing data, which I'm not going to try to
>> sort out here as I don't understand it all.  There are may types of
>> printable surfaces on the top of the disc, for automated printing of
>> labels.  Lightscribe and inkjet surfaces are examples.  These usually
>> increase the cost.  I don't need them.  For my purposes, writing on
>the
>> disc with a DVD safe sharpie pen is enough.  Based on my reading,
>double
>> layer discs suffer more compatibility problems so I'm not considering
>> them.  Double sided discs are much more likely to get scratched and
>> cannot easily be labeled so I'm not considering them.  It is actually
>> very hard to figure out all the DVD media options, but I've selected
>a
>> few to look at.  Prices are Amazon, if they sell the product in
>question.
>>
>> Gotta account for the case for the DVD, about $ 0.25 each.
>>
>> The DVD-R format seems to be the most universally compatible.  I'm
>not
>> trying to list all discs here, just get representative prices.  Of
>> conventionally constructed discs, Verbatim seems to have a good
>> reputation, but there are varying reviews.  Longevity estimates for
>> conventional disc media range from 5 years anecdotal to 50 years from
>> the factories.  I will say that I have some 5 year old Maxell DVD-R's
>> that are still readable, but I will probably copy them to archival
>discs.
>>
>> VERBATIM 94852 DVD-r media 8x 4.7gb 50-pk spindle (shiny silver) - $
>19
>> / 50 = $ 0.38 / disc.  Cost with case (cwc) = $ 0.63 / disc
>> Verbatim Shiny Silver DataLife Plus 8x DVD-R Spindle, 50 Discs - also
>$
>> 19 / 50
>>
>> It seems that JVC Taiyo Yuden is the top of the heap in terms of
>quality
>> for conventional media.  Taiyo Yuden bought part of JVC as I
>understand
>> it and they're now shipping with JVC part numbers and packaging. 
>Many
>> people still know the Taiyo Yuden name and so that's usually listed
>in
>> the product listings.  Apparently, some TY discs themselves are
>> unlabeled, so if you want TY, make sure you get TY.
>>
>> 100 Taiyo Yuden/JVC 8X DVD-R 4.7GB Silver Thermal Lacquer - $ 29 /
>100 =
>> $ 0.29 / disc.  cwc = $ 0.54 / disc  Better product, cheaper price. 
>Cool.
>>
>> Still, I don't consider these discs long term storage.  The next step
>up
>> is archival grade discs, which are much more expensive.  Archival
>grade
>> discs usually have a solid gold layer to minimize corrosion and
>maximize
>> reflectivity.  Some have gold only.  Some have gold and silver, which
>is
>> cheaper - less gold, more silver.  They also usually have an extra
>hard
>> coating to prevent scratches.  Conventional DVD's are insanely easy
>to
>> scratch.
>>
>> Verbatim 95355 UltraLife 4.7 GB 8x Gold Archival Grade DVD-R, 50-Disc
>> Spindle - $ 93 / 50 = $ 1.86 / disc.  cwc = $ 2.11 / disc
>>
>> My research tells me that MAM-A / Mitsui is state of the art in
>archival
>> grade discs.  They have lots of data on their website.  Their gold /
>> silver DVD discs are projected to last 83 years.  Their gold DVD
>discs
>> are projected to last 100 years.  And their gold CD discs are
>projected
>> to last 300 years.  Also, note the same risk as above in buying
>> unlabeled discs.
>>
>> http://www.mam-a.com/
>>
>> MAM-A Mitsui Gold Archive 8x DVD-R with no logo in bulk - 50 Count =
>$
>> 117 / 50 = $ 2.34 / disc.  cwc = $ 2.59 / disc  (100 year discs)
>> MAM-A 8x SILVER PLUS GOLD DVD-R Branded Gold Surface 50pk Beehive
>with
>> Hard Coat - $ 93 / 50 = 1.86 / disc.  cwc = $ 2.11 / disc  (83 year
>> discs)  (factory price)
>> MAM-A 84019 Silver plus Gold DVD-R 16x 4.7GB MAM-A Logo 50pk Spindle
>-
>> 84019 - $ 80 / 50 = $ 1.60 / disc.  cwc = $ 1.85 / disc  (83 year
>> discs)  (dealer price)
>>
>> This last one is probably the best compromise between price and
>longevity.
>>
>> Finally, if you want longevity for 1000 years, there's m-disc, as
>> mentioned previously.  This is overkill for what I'm doing, but here
>are
>> the prices.
>>
>> M-DISC DVD+R 4X Cake Box 50 Pack - $ 136 / 50 = $ 2.72 / disc.  cwc =
>$
>> 2.97 / disc.  That's only $ 0.38 / disc more than the 100 year discs.
>> Not to bad if you need that kind of lifespan.
>>
>> So, the best I can do for an archival grade disc is $ 1.85 / disc.
>>
>> Now, what about the HDD option.  Here's a 3 TB drive I found on
>Amazon.
>>
>> Western Digital Caviar Green 3 TB SATA III 64 MB Cache Bare/OEM
>Desktop
>> Hard Drive - WD30EZRX - $ 140
>>
>> Let's say I want two for redundancy, since this is NOT archival
>media.
>> I would definitely spinrite this thing a couple of times / year to
>> refresh the magnetic fields.  That process would probably take 5
>days.
>> 2 HDD - $ 280
>>
>> How about a nice dual drive case with fan.
>>
>> Vantec Dual 3.5-Inch SATA to USB 3.0 and eSATA with JBOD/RAID 0/1
>> External Hard Drive Enclosure (NST-400MX-S3R) - $ 83
>>
>> So the total cost is - $ 363
>>
>> And we can store 3000 GB / 4.7 GB per hour = 638 hours of video
>> approximately equivalent to 638 DVD's.
>>
>> So, cost per equivalent DVD works out to - $ 363 / 638 = $ 0.57 /
>"disc"
>> or hour.  This is slightly more than the cost of cheap DVD's when
>cost
>> of cases is accounted for but substantially less than the cost of
>> archival DVD's.
>>
>> So, here's how I see my choices.
>>
>> - 83 year archival DVD's
>>
>>     * relatively low up front cost for 50 pack
>>     * substantially larger long term cost
>>     * some additional cost for storage rack, etc.  (and where do I
>put it)
>>     * discs are easy to access and play in dvd player
>>
>> - two big hard drives
>>
>>     * relatively high up front cost
>>     * substantially lower long term cost
>>     * minimal space required
>>     * movies are harder to access since I have to hook up computer to
>tv
>> and start player, etc.
>>
>> There is the possibility of online storage.  Storing 5 GB on Amazon
>S3
>> will cost me $ 0.75 / month for 1 hour of video.  It will cost me
>about
>> $ 0.75 each time I retrieve it.  Given the quantity of data, not very
>> practical.
>>
>> None of these choices are really great for me.  What I may do is to
>> select the best 50 - 100 hours of source video and put them onto 83
>year
>> archival discs.  Even that would cost me $ 185.  I'm not sure it's
>worth
>> it.  If I'm comparing the HDD to the 83 year archival discs, the hard
>> drive becomes more economical than the discs once I store 196 hours
>of
>> video.  At this point, it seems there are no good (within my budget)
>> options to store 800 hours of video.  One other option is to convert
>> everything to cheap DVD's, then refresh it and recopy it every 5 - 10
>> years.  That doesn't sound too attractive either.
>>
>> PS, from a hardware perspective, the Hauppauge USB-Live2 analog video
>> digitizer looks cool.  Haven't tried it and don't know about it's
>> drivers or software.  I can get mdisc / bluray drives in the $ 50 -
>100
>> range.
>>
>> Let me know what you think, and whether I've left out anything major.
>> This took two hours to type and I didn't proofread it, so sorry for
>any
>> errors.
>>
>> Sincerely,
>>
>> Ron
>>
>>
>> On 10/15/2012 4:45 PM, JD wrote:
>> >> I also wanted to share a great way to track and catalog DVD's that
>> >> you've purchased.  It's an android app called MyMovies (free -
>limited
>> >> to 50 titles) and MyMovies Pro ($6 - unlimited).  It's ultra cool.
> Just
>> >>
>> > Hopefully, someone else has found a fantastic Linux video editor
>and
>> will be
>> > able to let us all know about it.
>> >
>> >
>> >
>> > Previously I used a F/LOSS web server + back-end DB to catalog DVDs
>and
>> > recordings.  Over time, that became too much effort and I didn't
>like
>> that the
>> > program forced me to a specific DB and the export was only to HTML
>> files.  Tried
>> > a Java desktop client too, but it was toooo damn slow.  It didn't
>play
>> nice on
>> > my network.
>> >
>> > Ended up using the time-proven 'ls -Rl' technique with numbered
>DVDs.
>>  With a
>> > large number of data DVDs holding TV recordings in xvid/avi or
>> h.264/mkv, a
>> > quick 'egrep -i' searches by title for the specific numbered disk
>> containing the
>> > recorded show/movie.  As a CLI person, the most-used desktops all
>have a
>> script
>> > that will ssh over to the main box with all the dvd-0xyz.txt files
>and
>> perform
>> > the query.  For more information, XBMC has the catalog with actors,
>> directors,
>> > year, genre, title,  etc ....  grep is just so useful and text is
>so
>> easy to
>> > search.
>> >
>> > I love that the grep clearly returns the numbered data DVD I need
>to
>> load.
>> >
>> > As to converting VHS to MPEG or XVID or h.264, almost any $20 USB
>TV
>> adapter can
>> > record SD content to MPEG2.  Just pick from a MythTV compatible
>device
>> to be
>> > happiest long term.  If you want to record OTA HD content, I'd
>suggest
>> more
>> > effort for the selected hardware.  I love the HD-Homerun network
>tuners.
>> Those
>> > work well with OTA except 1 channel that doesn't come in at my
>house. It
>> is
>> > hardly a major issue, since 69 other channels do come in OTA from
>> Marietta.
>> >
>> > For finding commercials and editing them out, I searched for
>Linux-based
>> > solutions for years and finally gave up. Comskip does run nice
>under
>> WINE, I've
>> > never found a Linux-based editor that will accept the comskip
>created
>> cut files
>> > (there's like 6 different formats that comskip will create) as the
>> starting
>> > point for where to cut commercials.  Comskip is about 90% accurate.
>>  Comskip can
>> > output cut marker files that XBMC and mplayer will honor without
>> altering the
>> > source video file. That can be handy when you just want to skip
>> commercials, but
>> > not waste time actually modifying the file contents.
>> >
>> > Instead of fighting it, I bought - yes, purchased a Windows tool
>called
>> > VideoRedo TV Suite.  It appears to use comskip code internally to
>create
>> cut
>> > markers.  There's a point where $50 is worth it to me.  That $50
>program
>> has
>> > been heavily automated here to generate the cut marker files
>overnight.
>> It takes
>> > about 45 seconds of human effort to confirm the cut locations and
>> request the
>> > modified file be saved.  If you are willing to live with 90%
>accurate
>> cuts, it
>> > could all be automated.  Networks have changed up the markers
>commonly
>> searched
>> > by VRDP and comskip to trick them into keeping commercials and
>removing
>> some of
>> > the desired content.  Older stations with less money like RTV are
>almost
>> 100%
>> > accurate with cut locations.
>> >
>> > At this point, my systems are almost automatic with just a few
>manual
>> > "confirmation" places.  WTV -->  MPEG2 -->  h.264/mkv with CC1 and
>CC3
>> closed
>> > captions included.  If there are multiple audio channels in the
>stream,
>> that can
>> > require special processing, but that usually happens with a few PBS
>> programs
>> > with audio tracks for the deaf or background-only audio.
>> >
>> > I've blogged about almost all of these solutions over the years.
>The
>> most recent
>> > articles are about building your own DB4 antenna for $20 or so that
>> actually
>> > beat some Antennas Direct models.
>> >
>> >
>> > And just to be clear, I also wimped out and use Windows7 to record
>TV.
>> It is
>> > just so easy to setup with free schedule data. That OS runs inside
>a
>> virtual
>> > machine under KVM and has been doing it nicely for over a year. 
>THAT
>> was not an
>> > easy thing to accomplish to get satisfactory results. Microsoft did
>> (does?) some
>> > things trying to protect Joe-end-user from doing foolish things,
>like
>> blocking
>> > recording directly to NAS storage.
>> >
>> > I suspect there are hundreds of years of knowledge lurking related
>to
>> this problem.
>> > _______________________________________________
>> > Ale mailing list
>> > Ale at ale.org
>> > http://mail.ale.org/mailman/listinfo/ale
>> > See JOBS, ANNOUNCE and SCHOOLS lists at
>> > http://mail.ale.org/mailman/listinfo
>> >
>> >
>>
>> --
>>
>> (To whom it may concern.  My email address has changed.  Replying to
>former
>> messages prior to 03/31/12 with my personal address will go to the
>wrong
>> address.  Please send all personal correspondence to the new
>address.)
>>
>> (PS - If you email me and don't get a quick response, you might want
>to
>> call on the phone.  I get about 300 emails per day from alternate
>energy
>> mailing lists and such.  I don't always see new email messages very
>> quickly.)
>>
>> Ron Frazier
>> 770-205-9422 (O)   Leave a message.
>> linuxdude AT techstarship.com
>>
>> _______________________________________________
>> Ale mailing list
>> Ale at ale.org
>> http://mail.ale.org/mailman/listinfo/ale
>> See JOBS, ANNOUNCE and SCHOOLS lists at
>> http://mail.ale.org/mailman/listinfo
>>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Ale mailing list
>Ale at ale.org
>http://mail.ale.org/mailman/listinfo/ale
>See JOBS, ANNOUNCE and SCHOOLS lists at
>http://mail.ale.org/mailman/listinfo


--

Sent from my Android Acer A500 tablet with bluetooth keyboard and K-9 Mail.
Please excuse my potential brevity.

(To whom it may concern.  My email address has changed.  Replying to former
messages prior to 03/31/12 with my personal address will go to the wrong
address.  Please send all personal correspondence to the new address.)

(PS - If you email me and don't get a quick response, you might want to
call on the phone.  I get about 300 emails per day from alternate energy
mailing lists and such.  I don't always see new email messages very quickly.)

Ron Frazier
770-205-9422 (O)   Leave a message.
linuxdude AT techstarship.com