Split large file into parts by record count 2

[johnherman]

Pardon me if this has been answered, but I did a search and did not achieve success. I want to split a large (multi-million record) file into separate files of one million each with the last piece containing the remainder. I thought about using head and tail, but I think awk will be faster and cleaner. Can someone help get me started?

-------------------------
The trouble with doing something right the first time is that nobody appreciates how difficult it was - Steven Wright

 

[motoslide]

I normally use dd for such tasks. However, this is best to just split a file which will later be re-assembled. If your records are of consistent length, dd could still be used.

$ dd if=bigfile of=file_sub1 bs=128 count=1000000
$ dd if=bigfile of=file_sub2 bs=128 count=1000000 skip=1000000
(and so forth)
This could be scripted in a loop, of course. You would need to do a small sample to get the correct Block Size (option "bs" above). For example, if your records are exactly 256 bytes long, the "bs" should probably be 257 to include the line terminator.

 

[p5wizard]

If you are on a unix/linux box, see if you have the split command to achieve this:

split -l 1000000 bigfile part

creates files partaa partab partac ... partzz

see manpage for more info

HTH,

p5wizard

 

[johnherman]

Thanks guys, never heard of either command before. Stars around - for p5wizard for supplying the solution used and for motoslide for another solution perhaps useful in other areas since dd also provides EBCDIC to ascii conversion capabilities.

-------------------------
The trouble with doing something right the first time is that nobody appreciates how difficult it was - Steven Wright