Quantcast Linux: Building Encoding Cluster - digitalFAQ.com Forums [Archives]
  #1  
04-12-2004, 11:28 AM
venkatk venkatk is offline
Free Member
 
Join Date: Jan 2004
Location: Planet Earth
Posts: 26
Thanks: 0
Thanked 0 Times in 0 Posts
Hi,

Anyone tried using Avisynth/CCESPT with wine for KDVD/KVCD's?

Linux purists, please excuse me!

I am planning to build an OpenMosix cluster with three P4/2.8GHz/512MB-DDR400 nodes and use it to encode KDVD/KVCD as an experiment. I want to use TMPGEnc or CCESPT for encoding with Avisynth frame server.

Am I wasting time trying to use Avisynth/CCE-SPT with wine?
__________________
venkat
Reply With Quote
Someday, 12:01 PM
admin's Avatar
Site Staff / Ad Manager
 
Join Date: Dec 2002
Posts: 42
Thanks: ∞
Thanked 42 Times in 42 Posts
  #2  
04-12-2004, 12:28 PM
ak47 ak47 is offline
Free Member
 
Join Date: Oct 2002
Posts: 168
Thanks: 0
Thanked 0 Times in 0 Posts
In my opinion there is two ways to do it, debain based (knoppix) or gentoo based. If you chose to debain based I would recommend using clusterknoppix at http://bofh.be/clusterknoppix/ it is a live cd and comes all complied for you. You can also use gentoo which would be my choice, and get a fully custom OS with what you want, since you making a video encoder cluster you most likely want to use gnome for a desktop environment since it uses less resources (there is only few versions of knoppix that use gnome and clusterknoppix is not one), but beware gentoo is one of the hardest distributor to install, since it fully custom. Either way your most likely going to install it on the hard drive, since live cd is probably a crappy idea for a cluster since cd-roms take lost of time to spin up and down making each node that connect to the main server slow down the system. Its more of the question of time if you want to learn Linux and how hard it is then choose gentoo which will give you better results since you OS will be custom, if you don't have much time or just don't want to go though a kernel panic fix, then choose clusterknoppix.

P.S. Avisynth doesn't work and don't use tmpgenc or windoz based encoders mencoder is replacing that, and here http://www.kvcd.net/forum/viewtopic.php?t=9074 is the forum on all the info you need on mencoder, and here shows why you might not need avisyth http://www.kvcd.net/forum/viewtopic.php?t=10242
__________________
Later ak
Reply With Quote
  #3  
04-12-2004, 01:02 PM
venkatk venkatk is offline
Free Member
 
Join Date: Jan 2004
Location: Planet Earth
Posts: 26
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by ak47
In my opinion there is two ways to do it, debain based (knoppix) or gentoo based. If you chose to debain based I would recommend using clusterknoppix at http://bofh.be/clusterknoppix/ it is a live cd and comes all complied for you. You can also use gentoo which would be my choice, and get a fully custom OS with what you want, since you making a video encoder cluster you most likely want to use gnome for a desktop environment since it uses less resources (there is only few versions of knoppix that use gnome and clusterknoppix is not one), but beware gentoo is one of the hardest distributor to install, since it fully custom.
Installing would not be a problem. Have been using Linux professionally from past 8 years. I would be using distro based on RHEL 2.1AS or RH7.3 as I use them for my work. Already, I have couple of boxes with Linux at home

Quote:
Originally Posted by ak47
]P.S. Avisynth doesn't work and don't use tmpgenc or windoz based encoders mencoder is replacing that, and here http://www.kvcd.net/forum/viewtopic.php?t=9074 is the forum on all the info you need on mencoder, and here shows why you might not need avisyth http://www.kvcd.net/forum/viewtopic.php?t=10242
Thanks for the inputs.

People on Doom9 tell Avisythn, CCESPT work with wine. I have successfully installed Avisynth, VirtualDubMod and CCESPT, but could not get to load the avs script.

I have never tried Mencoder. Will give it a try. I was trying to use a mix of Big3 and MA script for backing up my DVD's with the menu structure intact. I am not sure whether Mencoder will meet my all needs.
__________________
venkat
Reply With Quote
  #4  
04-12-2004, 03:16 PM
ak47 ak47 is offline
Free Member
 
Join Date: Oct 2002
Posts: 168
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
People on Doom9 tell Avisythn, CCESPT work with wine.
I did not know that, i tried wine and wineX a while ago with Avisythn and it didn't work, then later i read they were trying to make a linux version because wine does not support all the dll features windows does.

Quote:
Have been using Linux professionally from past 8 years.
Quote:
I have never tried Mencoder
Shame on you for never trying Mencoder j/k, its now the biggest talk in video encoding lately in my opinion, and its originally design for RH, I always used windows programs until i found out libavcodec now i can't stop using linux encoders.

Quote:
was trying to use a mix of Big3 and MA script for backing up my DVD's with the menu structure intact. I am not sure whether Mencoder will meet my all needs.
Well Mencoder doesn't support authoring of any sort but it will give you the great looking video files out there.
__________________
Later ak
Reply With Quote
  #5  
04-12-2004, 07:48 PM
venkatk venkatk is offline
Free Member
 
Join Date: Jan 2004
Location: Planet Earth
Posts: 26
Thanks: 0
Thanked 0 Times in 0 Posts
I have been consulting Linux and Open Source technologies. Have few Linux boxes with oodles of power at home which can be used for cluster. A dual Xeon 2.8GHz is on the way

Quote:
Originally Posted by ak47
Shame on you for never trying Mencoder j/k, its now the biggest talk in video encoding lately in my opinion, and its originally design for RH, I always used windows programs until i found out libavcodec now i can't stop using linux encoders.
I have been visiting this forum on/off from past couple of years. But, I have started to encode only couple of months back. I wanted to use tools where you get more support. Also, pls read my next paragraph.

Quote:
Originally Posted by ak47
Well Mencoder doesn't support authoring of any sort but it will give you the great looking video files out there.
The movies I backup are almost 210-240 minutes long and you cannot have two movies on one DVD-5. So, I would like to retain the menu and subtitles as I have space for them. Using Mencoder, you can still retain the menu structure provided there is only on PGC. Again, the movies I backup generally have PGC's with duplicate vobid's and it seems only CCE provides the chapters.
__________________
venkat
Reply With Quote
  #6  
04-12-2004, 08:39 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by venkatk
The movies I backup are almost 210-240 minutes long and you cannot have two movies on one DVD-5.
What
I put ~4 movies on each DVD, using KDVD.
Average of ~8 hours, at 704x480 (wide screen material)
At ~6 hours per DVD using KDVD, you really can't tell the difference from the original DVDs
Now with MEncoder, we'll raise that bar even further.
Look what I just did using MEncoder.
http://www.kvcd.net/payback.mpg.cut.8.mpg
That's 704x480 MPEG-2, audio 112Kbps(44.1KHz) DS2
No filters used
The complete movie (101 minutes) muxed is 795,419KB

-kwag
Reply With Quote
  #7  
04-12-2004, 09:15 PM
venkatk venkatk is offline
Free Member
 
Join Date: Jan 2004
Location: Planet Earth
Posts: 26
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by kwag
What
I put ~4 movies on each DVD, using KDVD.
Average of ~8 hours, at 704x480 (wide screen material)
At ~6 hours per DVD using KDVD, you really can't tell the difference from the original DVDs
Now with MEncoder, we'll raise that bar even further.
Look what I just did using MEncoder.
http://www.kvcd.net/payback.mpg.cut.8.mpg
That's 704x480 MPEG-2, audio 112Kbps(44.1KHz) DS2
No filters used
The complete movie (101 minutes) muxed is 795,419KB

-kwag
Please forgive me for making a stupid statement. It was uncalled for.
I have been able to put 2 hours onto one single disk.

In fact, the zeal to retain the menu structure forced me to consider using windows tools with wine as reauthoring would be easy.

The clip looks very very good even when you zoom 300%.

I will forget about original menu and start encoding with Mencoder. I should be able to churn out lot's of KDVD's quickly as I have oodles of power on my Linux box(s).

Thanks for making me see the light
__________________
venkat
Reply With Quote
  #8  
04-12-2004, 09:37 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Let us know about your encoding adventures with your Linux farm (rub it in, eh!)

-kwag
Reply With Quote
  #9  
04-12-2004, 09:59 PM
venkatk venkatk is offline
Free Member
 
Join Date: Jan 2004
Location: Planet Earth
Posts: 26
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by kwag
Let us know about your encoding adventures with your Linux farm (rub it in, eh!)

-kwag
Sure Kwag. I expect to have the Farm ready in a day or two as I need to recompile the kernel on all the live boxes with OpenMosix patches.

BTW, I have no experience with mencoder. I will be coming back to the forum for more help.
__________________
venkat
Reply With Quote
  #10  
04-12-2004, 10:03 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by venkatk
Sure Kwag. I expect to have the Farm ready in a day or two as I need to recompile the kernel on all the live boxes with OpenMosix patches.
What about downloading "Cluster Knoppix", which already has Mosix built-in
Quote:

BTW, I have no experience with mencoder. I will be coming back to the forum for more help.
Great

-kwag
Reply With Quote
  #11  
04-12-2004, 10:07 PM
ak47 ak47 is offline
Free Member
 
Join Date: Oct 2002
Posts: 168
Thanks: 0
Thanked 0 Times in 0 Posts
To anwser kwags question with a quote from venkatk.
Quote:
Installing would not be a problem. Have been using Linux professionally from past 8 years. I would be using distro based on RHEL 2.1AS or RH7.3 as I use them for my work. Already, I have couple of boxes with Linux at home
__________________
Later ak
Reply With Quote
  #12  
04-12-2004, 10:09 PM
venkatk venkatk is offline
Free Member
 
Join Date: Jan 2004
Location: Planet Earth
Posts: 26
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by kwag
What about downloading "Cluster Knoppix", which already has Mosix built-in
I am considering "Cluster Knoppix" as cluster server. But couple of the Linux boxes are live servers which are accessed by my clients. So, I will be installing a new kernel with OpenMosix patches.
__________________
venkat
Reply With Quote
  #13  
04-12-2004, 10:25 PM
venkatk venkatk is offline
Free Member
 
Join Date: Jan 2004
Location: Planet Earth
Posts: 26
Thanks: 0
Thanked 0 Times in 0 Posts
I forgot to tell one more reason why I was considering wine.

Wine will spawn a new process for each windows thread. So, You can distribute the Avisynth process to one node and encoder to one or two nodes depending on how many threads the encoder is using internally.

Edit: Checking out whether I can use one mencoder instance for pre-processing and other for encoding?
__________________
venkat
Reply With Quote
  #14  
04-12-2004, 10:28 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Consider Cluster Knoppix, and also consider XFS when creating the partitions. To me, XFS is probably the best file system in existence.
EXT3 is good, until you fill up a partition, and then your system will crash (reported to my by a friend, who uses Linux on DNS/FTP servers at the corporate level )
I also have been playing with Linux since ~1992, but I use FreeBSD professionally (Since 1997) for all my customers
Although, I have my eyes focused on Linux kernel 2.6, which seems to match (or exceed!) FreeBSD's performance.
I still have to run some tests, though.

-kwag
Reply With Quote
  #15  
04-12-2004, 10:35 PM
venkatk venkatk is offline
Free Member
 
Join Date: Jan 2004
Location: Planet Earth
Posts: 26
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by kwag
Consider Cluster Knoppix, and also consider XFS when creating the partitions. To me, XFS is probably the best file system in existence.
EXT3 is good, until you fill up a partition, and then your system will crash (reported to my by a friend, who uses Linux on DNS/FTP servers at the corporate level )
I also have been playing with Linux since ~1992, but I use FreeBSD professionally (Since 1997) for all my customers
Although, I have my eyes focused on Linux kernel 2.6, which seems to match (or exceed!) FreeBSD's performance.
I still have to run some tests, though.

-kwag
Never had problems with EXT3 till now. Generally I use mix of ext3 and Reiser depending on needs. For example. caching servers will perform better with Reiser than EXT3.

I have used FreeBSD and OpenBSD professionally to a limited extent. Over here Linux is very popular and It pays to be a Linux consultant
__________________
venkat
Reply With Quote
  #16  
04-12-2004, 11:03 PM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by venkatk
Over here Linux is very popular and It pays to be a Linux consultant
Yes, that's for sure.
Here too
http://www.thelinuxreview.com/ldp/se...onsult_id=3252

-kwag
Reply With Quote
  #17  
04-13-2004, 10:37 AM
ak47 ak47 is offline
Free Member
 
Join Date: Oct 2002
Posts: 168
Thanks: 0
Thanked 0 Times in 0 Posts
venkatk wrote:
Quote:
Over here Linux is very popular and It pays to be a Linux consultant
Definitely thats why I decided to go into that career path, though I still am in HS. But if you know some parts of your software is supported and funded by certain government agencies (military, nasa, and others) than it has to be good.

Kwag wrote:
Quote:
To me, XFS is probably the best file system in existence.
EXT3 is good, until you fill up a partition, and then your system will crash
I am not saying your wrong because i never tried that one yet, since i stick to ones most people write programs thats built for, but here is Reiser4 http://www.namesys.com/v4/v4.html is defiantly taking a lead on the benchmark vote, though still being test therefore unstable, it is still 4x faster than NTFS and 2x faster than ext3.
__________________
Later ak
Reply With Quote
  #18  
04-13-2004, 10:50 AM
kwag kwag is offline
Free Member
 
Join Date: Apr 2002
Location: Puerto Rico, USA
Posts: 13,537
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by ak47
but here is Reiser4 http://www.namesys.com/v4/v4.html is defiantly taking a lead on the benchmark vote, though still being test therefore unstable, it is still 4x faster than NTFS and 2x faster than ext3.
ReiserFS is very good, but still behind XFS in production environments.
XFS is Silicon Graphics file system, which has been used in production for over 10 years in their IRIX Operating System.
Silicon Graphics produces high end machines, used for heavy duty graphics/animation/movie productions.
So it's a proven filesystem, and ReiserFS is still relatively new, so I wouldn't put my client's data on it yet

-kwag
Reply With Quote
  #19  
04-13-2004, 11:22 AM
venkatk venkatk is offline
Free Member
 
Join Date: Jan 2004
Location: Planet Earth
Posts: 26
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by kwag
ReiserFS is very good, but still behind XFS in production environments.
XFS is Silicon Graphics file system, which has been used in production for over 10 years in their IRIX Operating System.
Silicon Graphics produces high end machines, used for heavy duty graphics/animation/movie productions.
So it's a proven filesystem, and ReiserFS is still relatively new, so I wouldn't put my client's data on it yet

-kwag
Each file system has its own merits and drawbacks. As Karl pointed out, even I would not trust the Reiser with mission critical data. But, I use Reiser extensively on my caching servers as it gives better performance when you have large number of directories and small files.

To be frank Reiser has never been a problem. I am very conservative when it comes using new technologies in production. I still use Redhat 7.3 or my own distro forked from RHEL 2.1AS sources
__________________
venkat
Reply With Quote
  #20  
04-15-2004, 10:03 AM
venkatk venkatk is offline
Free Member
 
Join Date: Jan 2004
Location: Planet Earth
Posts: 26
Thanks: 0
Thanked 0 Times in 0 Posts
It seems mencoder does't support SMP . So, cluster will not help.
I am exploring other options. BTW, the motherboards I had ordered for building cluster were damaged in transit and I have to wait for another day.

Meanwhile, have a look at this http://www.linuxjournal.com/article.php?sid=7126

Also this http://www.cs.wisc.edu/condor/
__________________
venkat
Reply With Quote
Reply




Similar Threads
Thread Thread Starter Forum Replies Last Post
64k NTFS cluster size worth it for video processing? cweb Computers 5 05-03-2006 08:46 AM
EMPIRE STATE BUILDING glänzend Off-topic Lounge 0 10-01-2004 12:23 PM
Linux: Can desktop environments effect your encoding? ak47 Computers 1 01-28-2004 05:56 PM
Building a home entertainment Computer m0rdant Players, DVRs, Media Centers 1 01-02-2004 06:04 PM
Linux: Redhat Linux is dead, long live Fedora Core 1 jorel Computers 1 09-28-2003 04:05 PM

Thread Tools



 
All times are GMT -5. The time now is 09:11 AM  —  vBulletin © Jelsoft Enterprises Ltd