Editing a 16GB file 1

[zeland]

I've got a 16GB YUV video file that has a corrupted header. I'm trying to manually fix the header by editing it by hand but none of the editors (vi/vim, gedit, hexedit, etc) seem to be able to handle a file that big. Any recommendations?

--== Anything can go wrong. It's just a matter of how far wrong it will go till people think its right. ==--

 

[stevexff]

Get the header in a file - use head -10 corrupt.yuv > header.yuv to copy the first bit to another file. You might have to try different values of -n.

Then you can look at it with an editor. Figure out what's wrong with it, and fix it. Trim off any bits that aren't part of the header.

Use head, tail, sed, awk, or perl depending on your preference to catenate the new fixed-up header with the original file minus its broken header. Because you are using stream editors, they *shouldn't* balk at processing such a large file.

Steve

[small]"Every program can be reduced by one instruction, and every program has at least one bug. Therefore, any program can be reduced to one instruction which doesn't work." (Object:erlDesignPatterns)[/small]

 

[zeland]

I've actually identified the corruption in the header before posting using head -n 1 video.yuv. I was actually considering using sed, however, I don't have another 16GB of disk space to spare.

--== Anything can go wrong. It's just a matter of how far wrong it will go till people think its right. ==--

 

[zeland]

OK. I solved my problem by filtering my video file through sed and straight into my encoder without creating and intermediary file. Why I didn't think of that earlier is beyond me.

Anyway, I'm still curious if there are any editors out there that can handle files of this size. I remember way back in 92, Norton Utilities suite had a utility called disk edit or some thing like that which allowed you to edit files by changing the data on a per cluster basis. It was very efficient when editing large files as the editor loaded and saved data in 1k blocks (or what ever size you used to format your HDD with). Do we have anything like that in Linux?

--== Anything can go wrong. It's just a matter of how far wrong it will go till people think its right. ==--

 

[stevexff]

Tricky. It's so big you can't even burn it to a DVD. Short of plugging in a big USB disk drive to use as temp space, I'm not sure how you'd get around it.

Steve

[small]"Every program can be reduced by one instruction, and every program has at least one bug. Therefore, any program can be reduced to one instruction which doesn't work." (Object:erlDesignPatterns)[/small]

 

[thedaver]

Could you have used a binary or binhex editor to actually alter the file's sectors on disk? Rather dicey, but probably a means to get this simple job done. And, no, I don't know a product by name....

D


D.E.R. Management - IT Project Management Consulting

http://www.dermanagement.com/

 

[zeland]

I did this:

cat video.yuv | sed s/"YUV4MPEG2 W720 H480 F29969999:1000000 Ip A47244640276:0"/"YUV4MPEG2 W720 H480 F30000:1001 Ip A4:3"/ | yuvcorrect -v 0 -Y CONFORM | mpeg2enc -f 8 -F 4 -q 4 -o vid.m2v

The resulting video was about 1.8GB which fits the 3GB of free space that I had left.

--== Anything can go wrong. It's just a matter of how far wrong it will go till people think its right. ==--

 

[zeland]

Right after my last post I stumbled across this:

http://lde.sourceforge.net/

It claims to be something like Norton disk edit. Will need to try this out some day.

--== Anything can go wrong. It's just a matter of how far wrong it will go till people think its right. ==--