join - outputting too few lines

[duncdude]

Why doesn't this work (properly) - can anyone help me?

users.txt
jdoe
cpurcell
ltorvalds

shadow.txt
hjablome:$1$89ekDhxk$MMKcoULbeURwAV4owiqB/:12585:0:99999:7:::
jdoe:$1$jn53vLCl$P/ZtXVPEWvIb518O3vr.:12585:0:99999:7:::
cpurcell:$1$gggLxru$XATg1YNJGi2IuiYCZSaZe.:12587:0:99999:7:::
nobody:$1$.AsmP4tL$Ebg1zADlMyv/w7TlhCqO0:12591:0:99999:7:::
ltorvalds:!!:12600:0:99999:7:::

[red][tt]join -t: users.txt shadow.txt[/tt][/red]

it only outputs:-

[tt]jdoe:$1$jn53vLCl$P/ZtXVPEWvIb518O3vr.:12585:0:99999:7:::
cpurcell:$1$gggLxru$XATg1YNJGi2IuiYCZSaZe.:12587:0:99999:7:::[/tt]

... ltorvalds is missing - and I don't understand why?


Kind Regards
Duncan


Kind Regards
Duncan

 

[PHV]

Even if the two files are sorted by the join key ?
man join

Hope This Helps, PH.
Want to get great answers to your Tek-Tips questions? Have a look at FAQ219-2884 or FAQ222-2244

 

[duncdude]

PHV - thank you! working a treat now. didn't realise they needed to be sorted... seems kind of weird to me that they need to be but i'm sure there is a very valid reason

is this the easiest way to do the task in one go?

[red][tt]sort users.txt -o users.txt; sort shadow.txt -o shadow.txt; join -t: users.txt shadow.txt[/tt][/red]


Kind Regards
Duncan

 

[PHV]

is this the easiest way to do the task in one go?
1) The fgrep (grep -F) way:
fgrep -f users.txt shadow.txt
2) The awk way:
awk -F':' 'BEGIN{while((getline<"users.txt")>0)++a[$1]}a[$1]' shadow.txt

Hope This Helps, PH.
Want to get great answers to your Tek-Tips questions? Have a look at FAQ219-2884 or FAQ222-2244

 

[duncdude]

PHV

They are both fantastic solutions - thank you!

I have just bought a book on sed and awk and i'm preparing myself for alot of head scratching...

But please could you help me with your examples

I am assuming that the fgrep example (which I understand to be the same as grep -f) is using the text from the users.txt file - but only up to the first colon? Why does it do that?

Your awk example is rather more intimidating. I understand the principle - please can you explain the ++a[$1]}a[$1] section


Kind Regards
Duncan

 

[duncdude]

yep

Hi Paul - these will all do the job - obviously WITHOUT Perl

1st is clumsy, 2nd is clever (but i'm not quite sure - yet - how it works), 3rd is VERY clever

I only gave this example on the Perl forum because I imagine that we all need inspiring - I did not have the first clue that a problem like this could be solved on the command line with such a neat, one-line solution, until quite recently. I love Perl, but the awk solution is surely far more sensible - if the option is available. What do you think?

[tt]sort users.txt -o users.txt; sort shadow.txt -o shadow.txt; join -t: users.txt shadow.txt

fgrep -f users.txt shadow.txt

awk -F':' 'BEGIN{while((getline<"users.txt")>0)++a[$1]}a[$1]' shadow.txt[/tt]

many thanks to PHV - credit where credit is due!


Kind Regards
Duncan

 

[PHV]

the same as grep -f
I would say that fgrep is the same as grep -F
but only up to the first colon?
Unfortunately, no.
If in users.txt you have a line like cpur then a hit will be found in shadow.txt with cpurcell.
can you explain the ++a[$1]}a[$1] section
awk -F':' '
Launch awk with a colon ) as field separator
BEGIN{while((getline<"users.txt")>0)++a[$1]}
Before cycling thru the input file, read all the lines in users.txt populating an associative array (a) indexed by the first field ([$1] ie a username) with a counter (++)
a[$1]
This is a shorthand for a[$1]!=0{print}
If the first field in the current line exists in the array with a non-zero counter (ie exists in users.txt) then print the whole inpit line
' shadow.txt
The input file for the awk program is shadow.txt

Hope This Helps, PH.
Want to get great answers to your Tek-Tips questions? Have a look at FAQ219-2884 or FAQ222-2244

 

[duncdude]

thanks PHV!


Kind Regards
Duncan