Script:Resume a tar (and gzip)

I was compressing a huge text file into a tar.gz while doing about 10 other things this morning when my computer froze up. I desperately tried to get into the system monitor application and switching desktops (ctrl+alt+F5) with no luck. I had to actually use the power button 😦

7GB has already been written and I didn’t want to start over, so I just created this script to resume the creation of tar files.

#!/bin/bash

if [ $# -eq 0 ]; then
echo $0 [INFILES] [OUTFILE]
exit
fi
INFILES=$1
OUTFILE=$2
SIZE=”$(wc -c < $OUTFILE)”
tar -cvvz –to-stdout “$INFILES” | tail -c +$(($SIZE+1)) >> “$OUTFILE”

echo “${OUTFILE} creation completed!”

It seems to have worked just fine. I can’t see any errors in the file, but it was simply a large CSV. Give it a shot with different file types and report your successes and failures in the comments!

Quick Tutorial: Resizing/Moving Partitions without Losing Data

More and more people are dual booting or have more than one partition. Sometimes we might decide we wish we had sized our partitions differently after we have installed the OS (s). Well, as long as you have some time, you can do so without losing data. And, you can even do it with a GUI. All you need is a blank DVD or a USB flash drive. 

The first step is getting yourself a copy of Linux. The easiest would probably be Ubuntu or BackBox for most people. Here’s what to do: 

1. Download a Linux ISO. I’d use the torrent as it’s much faster. Ubuntu is at http://torrent.ubuntu.com:6969/file?info_hash=i%1F%D3%8F%97%98%00%7F%9D%BF%E3%24%11%0E%F9%8A%D4%80%F6p and BackBox is at https://backbox.org/download (you can set the donation to $0)
2. Either burn the ISO to a DVD or use these instructions to make a bootable flash drive. I’d definitely go with the flash drive if you can boot USB 3.0.
3. Put the DVD or flash drive in your computer and reboot. 
4. You may need to hit F12 or some other key to choose your boot device. Boot off the DVD or flash drive you just created. 
5. Once in Linux, use the task bar and search for “gparted”. Start the GParted program. 
6. Use the drop down to select the physical device your partition is on. 
7. Unmount the partition using the right click menu if it has a key icon next to it. 
8. Right click the partition and select “Resize/Move”
9. Now, note if the partition you hope to grow is above or below the partition you will shrink or take space from. 
10. If you are growing the partition on top, right click the partition on the bottom and choose “Resize”. If you are growing the lower partition, right click the top partition and choose “Resize”.
11. Resize your partition using the 3 numbers. If you are resizing the bottom partition, then use the arrows on the top number to grow the space before. For the top, grow the number after. The middle number or partition size, will shrink as you grow the other numbers. 
12. When you are done, you should be left with free space above or below depending on the partition you chose. 
13. Now right click the partition you would like to grow and choose “Resize”.
14. Use the up arrow on the middle number, or partition size, to grow the partition. The free space will automatically be taken away. 
15. Click “Apply” and find something to do. Gparted will need to move around file chunks, so the more full the partition your shrank is, the longer this will take. 
16. After anywhere from 15 minutes to a few hours the process will complete. Reboot and remove your flash or USB. Boot normally now. 
17. If all went to plan, you will boot up with your now resized partitions. 

I hope this helped. Hold on to your DVD or USB drive. It can be used for many things and if it’s USB 3.0, it will probably run faster than your normal drive. 

Linux Commands: Formatting a CSV for Importing in MySQL

So I use phpMyAdmin most of the time. I am working on a project (http://www.kubisec.com / http://kubisec.mywire.org) that requires I import millions of records into MySQL. The CSV import is pretty sensitive and on a lot of errors, it just gives up. I needed a sure way to make sure CSVs would import correctly. Fixing millions of lines manually was not an option.

I’m running BackBox Linux. It is basically Ubuntu with features from Kali and Tails. It has pen-testing tools and “anonymous” mode which sends all traffic through Tor. So this should work on most flavors and especially Ubuntu.

Here’s an example from the MySpace data dump:

358274297:em@il:no1believer:0x3C820C6E5C0A45D922BAC3A216F393BAC2646CC3:”
358274301:em@il:358274301:0x4F972B0E0284DD6A478A83E7CC3CB04DC7A3132C:”
358274303:em@il:358274303:0x85023BC72A13542002BDA3F31355C0BAD2767B38:”

It isn’t exactly CSV, but uses a “:”. That’s fine, this method will work with whatever separator you want. We want the email address, username, and the password hash (sha1). We are gonna use awk to pull out those fields and reprint what we want. Reprinting it ensures it is all formatted the same. It also ensures that we know the column count. Here’s the command:

awk ‘BEGIN{FS=”:”} {print $2″:”$3″:”$4}’ MySpaceDump/MySpace.part2.ab.csv > MySpace.formatted.part2.ab.csv

Here’s a breakdown of the command:

• awk is a tool/language for processing text
• BEGIN{FS=”:”} says this file is separated with “:”
• {print $2″:”$3″:”$4} says we want to print columns 2, 3, and 4
• {print $2″:”$3″:”$4} also says to print “:” between columns
• MySpaceDump/MySpace.part2.ab.csv is the input file
• > MySpace.formatted.part2.ab.csv says where to output

Our output file looks like this:

em@il:358274314:0x9233CD2EC86BC1F998285BE69CC997ACCF84565B
em@il:358274315:0xB56B1F4A6EBF893091F6BC0A91A498F18B796AA4
em@il:358274316:0x15524616E0D8C2C83FCD0F2056A6CD140A050F6C
em@il:randooman:0x429AAB47B254D09CF5CBA9653466C3904DA5DD61

This will get rid of problems from lines with more or less columns than the rest of the file, blank lines, spaces in the file, lines formatted wrong, and other stuff that will break a CSV import.

Lot’s of files are going to have little, different issues when importing into MySQL. This command will hopefully catch all of them. That said, you should write down commands as you go. That way, you can make a script for next time.

 

Linux Command: Bulk Renaming, Adding a Prefix/Suffix to Multiple Files

I split a large file into 1GB chunks. Since I piped half the file into split, I didn’t get to choose the file names. I wanna use a prefix that I recognize later. I also wanted to add a suffix of .csv so I remembered what the contents are. Here’s what I did:

backbox:~/Storage02/MySpaceDump$ ll -h
total 18G
drwxr-xr-x 6 dan dan 176K Oct 16 21:00 ./
drwxr-xr-x 14 dan dan 16K Dec 31 1969 ../
-rw-r–r– 1 dan dan 1.0G Oct 16 20:49 xaa
-rw-r–r– 1 dan dan 1.0G Oct 16 20:50 xab
-rw-r–r– 1 dan dan 1.0G Oct 16 20:50 xac
…

backbox:~/Storage02/MySpaceDump$ rename ‘s/xa/MySpaceDump.half2.xa/g’ xa*
dan@backbox:~/Storage02$ ll -h
total 18G
drwxr-xr-x 6 dan dan 176K Oct 16 21:51 ./
drwxr-xr-x 14 dan dan 16K Dec 31 1969 ../
-rw-r–r– 1 dan dan 1.0G Oct 16 20:49 MySpace.part2.xaa
-rw-r–r– 1 dan dan 1.0G Oct 16 20:50 MySpace.part2.xab
-rw-r–r– 1 dan dan 1.0G Oct 16 20:50 MySpace.part2.xac
…

backbox:~/Storage02/MySpaceDump$ zsh

backbox ~/Storage02/MySpaceDump
% autoload zmv

backbox ~/Storage02/MySpaceDump
% zmv -w ‘MySpace.part2.*’ ‘MySpace.part2.$1.csv’

backbox ~/Storage02/MySpaceDump
% ls -al
total 18325520
drwxr-xr-x 2 dan dan 16384 Oct 16 22:15 .
drwxr-xr-x 6 dan dan 180224 Oct 16 21:52 ..
-rw-r–r– 1 dan dan 1073741772 Oct 16 20:49 MySpace.part2.aa.csv
-rw-r–r– 1 dan dan 1073741806 Oct 16 20:50 MySpace.part2.ab.csv
…

Here’s what we did:

1. Used the rename command to bulk rename the files with a prefix of MySpace.part2.
2. Hopped into a Z shell by saying zsh
3. Enabled the zmv commad with autoload zmv
4. Used zmv to add .csv to the end of the file

These files are for http://www.kubisec.com / http://kubisec.mywire.org. Next I need to clean the files up so I can smoothly import them into MySQL. Check out my next post to see how I did this.

Linux Command: Split Half of Large File by Size/Lines Into Alternate Directory

~/Storage02$ wc -l ~/Storage01/Myspace.com.txt
360213049

~/Storage02$ calc 360213049/2
    180106524.5

~/Storage02$ tail -n +180106524 ~/Storage01/Myspace.com.txt | split -C 1G

This command will take the file Myspace.com.txt (the MySpace data dump) in the folder Storage01 and split it into files that are about 1GB in size in the Storage02 folder. The files will be just under 1GB. This is because the -C switch will break the file up by line. That means the files will fit as many full lines in 1GB as possible without going over and without breaking a line apart.

Storage02 is on a flash drive with a max file size of 1GB and Storage01 does not have enough room to fit another copy of the 33GB file. To use this command, first cd into the directory that you want to split the file into. count the lines using wc -l. Quickly get the half way point using calc. If you don’t have calc, say sudo apt-get install calc. From the directory you want to write to, you use tail -n to print all the lines after the half way point and pipe that into split.

This is what we accomplish in order:

1. Count the lines in our large file
2. Divide the line count by 2 to get the half-way point
3. Use tail to spit out the last half of the file
4. Pipe the last half into split
5. Split the file into 1GB pieces into a different directory

From here I plan to put the file into a MySQL database so that I can use it on a project I’m working on (http://www.kubisec.com/ http://kubisec.mywire.org)