Automated login using commandling in ubunto

[lhg1]

Hi

I would like to use an script to automaticly login to a webpage, and retrive the webpage to a txt file.

I tryed to get it to work using curl

curl --user name:password [URL unfurl="true"]http://www.url.com[/URL] -v >> webpage.log

But this does not work, the page is buield in asp and has a form with the login and Password.

I have been trying to read about how to save a login and password, maybi just the cookie, but haven't found anything that helps yet.

Regards
Lhg

 

[feherke]

Hi

Your code will work if the site uses HTTP authentication. If the username and password has to be entered in a HTML [tt]form[/tt], most probably there is no HTTP authentication behind it. ( No idea whether is possible in ASP. )

Probably you will have to make two requests with [tt]curl[/tt] : first to send the [tt]form[/tt] data for authentication, then to download the page. And most important, you have to make sure the 2[sup]nd[/sup] requests sends back the cookies received on the 1[sup]st[/sup] request.

By the way, I used to do such things with twill. Take a look, is not too complicated.


Feherke.
feherke.ga

 

[lhg1]

Hi

I just curl the page and this seems to be the password box.

<div style="width: 180px; margin: 3px 3px 7px 3px;"><table cellspacing="0" cellpadding="0" style="width: 100%;"><tr class="mtop"><td><div style="margin: 10px 3px 0px 10px;"><span class="
TitleHaed">Login <img src="/App_Themes/Normal/Images/Window/Elements/rightarrow.gif"><img src="/App_Themes/Normal/Images/Window/Elements/rightarrow.gif"><img src="/App_Themes/Normal/Images/Windo
w/Elements/rightarrow.gif"></div></td></tr><tr class="mmid"><td><div style="margin: 3px 10px 0px 10px">
<TABLE cellSpacing=3 cellPadding=3 width="100%" border=0>
  <TR vAlign="top">
    <TD align="center" width="55">
            <div style="float: left; margin:5px"><img id="ctl00_mnuLeft1_mSignin_mnuSignin_Image1" src="App_Themes/Normal/Images/Window/Pictures/woman.gif" border="0" /></div>
            <img id="ctl00_mnuLeft1_mSignin_mnuSignin_imgBlank1" src="Images/blank.gif" height="3" width="100%" border="0" />
            <A href="/forgotpassword.aspx">Glemt dit kodeord?</A> </TD>
    <TD>
        <SPAN class="MenuText">Brugernavn</SPAN>&nbsp; <br />
        <input name="ctl00$mnuLeft1$mSignin$mnuSignin$Username" type="text" maxlength="30" id="ctl00_mnuLeft1_mSignin_mnuSignin_Username" class="Text" onkeypress="return clickButton(event,'ctl00
_mnuLeft1_mSignin_mnuSignin_loginButton')" /><BR>
        <SPAN class="MenuText">Kodeord &nbsp;<br />
        </SPAN>
        <input name="ctl00$mnuLeft1$mSignin$mnuSignin$Password" type="password" maxlength="30" id="ctl00_mnuLeft1_mSignin_mnuSignin_Password" class="Text" name="password" onkeypress="return clic
kButton(event,'ctl00_mnuLeft1_mSignin_mnuSignin_loginButton')" /><BR>
        <span class="MenuText"><input id="ctl00_mnuLeft1_mSignin_mnuSignin_Remember" type="checkbox" name="ctl00$mnuLeft1$mSignin$mnuSignin$Remember" checked="checked" /><label for="ctl00_mnuLef
t1_mSignin_mnuSignin_Remember">Husk mig</label></span><BR>
         <img id="ctl00_mnuLeft1_mSignin_mnuSignin_imgBlank2" src="Images/blank.gif" height="3" width="100%" border="0" />
        <input type="image" name="ctl00$mnuLeft1$mSignin$mnuSignin$loginButton" id="ctl00_mnuLeft1_mSignin_mnuSignin_loginButton" src="App_Themes/Normal/Images/Buttons/login.gif" onclick="javasc
ript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(&quot;ctl00$mnuLeft1$mSignin$mnuSignin$loginButton&quot;, &quot;&quot;, true, &quot;signin&quot;, &quot;&quot;, false, false))" bor
der="0" />
     </TD>
  </TR>
</TABLE>

     </TD>
  </TR>

twill seems like a good idea, I have a littel problem transfering the eksampel to my retrived code.

setlocal username <your username>
setlocal password <your password>

go [URL unfurl="true"]http://www.slashdot.org/[/URL]
formvalue 1 unickname $username
formvalue 1 upasswd $password
submit

code 200     # make sure form submission is correct!

do I use the ID for the formvalue
id="ctl00_mnuLeft1_mSignin_mnuSignin_Username"
id="ctl00_mnuLeft1_mSignin_mnuSignin_Password"

Regards
Lhg

 

[feherke]

Hi

Well, I always used [tt]name[/tt]s in twill commands. Fortunately those [tt]input[/tt]s have [tt]name[/tt] attributes, so use those. Something like this :

[b]formvalue[/b] 1 "ctl00$mnuLeft1$mSignin$mnuSignin$Username" [i]your_username_here[/i]
[b]formvalue[/b] 1 "ctl00$mnuLeft1$mSignin$mnuSignin$Password" [i]your_password_here[/i]
[b]submit[/b]

Of course, this will not work if :
[ul]
[li]There are other [tt]form[/tt]s in the page before this one ( for example a search form in the page header, like we have here on Tek-Tips too ). Then you will have to adjust those 1 to match the login [tt]form[/tt]'s order number.[/li]
[li]The [tt]form[/tt] tag's [tt]action[/tt] attribute is empty or has a dummy value. Then you have to find out where the [tt]form[/tt] is actually sent and tell it to twill with [tt]formaction 1 real_url_here[/tt] before calling [tt]submit[/tt].[/li]
[/ul]


Feherke.
feherke.ga

 

[lhg1]

I'm having trouble getting this to work.

I tryed to change the formvalue but still I only get this message in twill-sh

Oops! Twill assertion error on line 1 of 'kandu.twill' while executing

>> formvalue 1 "ctl00$mnuLeft1$mSignin$mnuSignin$Username" l1977

no matching forms!

** UNHANDLED EXCEPTION: no matching forms!

this is the page I am trying to create a autologin script for.
kandu.dk

I'm not sure what you ment by the secound "will not work if"

Thanks
Lhg

 

[feherke]

Hi

Seems they have a "clever" firewall. By default [tt]twill[/tt] sends no [tt]User-Agent[/tt] HTTP request header and that triggers the firewall's suspicion. So it sends back an error page with no [tt]form[/tt]. Probably that is why you get that strange error message.

Add this somewhere before the [tt]go[/tt] command in your [tt]twill[/tt] script :

[b]agent[/b] "Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0"

Feherke.
feherke.ga

 

[lhg1]

Hi

Thanks, it allowed a bit more progress

I changed the login names to

formvalue 1 "ctl00_mnuLeft1_mSignin_mnuSignin_Username" L1977
formvalue 1 "ctl00_mnuLeft1_mSignin_mnuSignin_Password" larshg

But it is still not working, even thou I am not getting an error anymore

twill-sh kandu.twill3
>> EXECUTING FILE kandu.twill3
AT LINE: kandu.twill3:0
AT LINE: kandu.twill3:2
==> at

http://kandu.dk/

AT LINE: kandu.twill3:3
AT LINE: kandu.twill3:4
AT LINE: kandu.twill3:5
Note: submit is using submit button: name="ctl00$mnuLeft1$mSignin$mnuSignin$loginButton", value=""

AT LINE: kandu.twill3:7
--
1 of 1 files SUCCEEDED.

Any idears?

Thanks
Lhg

 

[feherke]

Hi

Ouch. There is a syntax error in the page's code : the password [tt]input[/tt] has two [tt]name[/tt] attributes. The browsers use and the server expects the 1[sup]st[/sup] one, but the [tt]twill[/tt] version I use handles the 2[sup]nd[/sup] one.

However is strange that your [tt]twill[/tt] not yelled for that. Maybe your version is better than mine, so try to continue.

Add this after the [tt]submit[/tt] command :

[b]code[/b] 200
[b]save_html[/b] [i]output_file_name.html[/i]

Then run the script and check the resulted output file.


Feherke.
feherke.ga

 

[lhg1]

Hi

I installed twill on an ubunto linux like this.

sudo apt-get install python-twill

In the HTML file I can se that the username is added, I cant se the code, I'm not sure if that is if its hidden, or if it didn't work.
And I seem not to have made the Login button

This is an image of the html page.

My current script

agent "Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0"

go [URL unfurl="true"]http://kandu.dk/[/URL]
formvalue 1 "ctl00_mnuLeft1_mSignin_mnuSignin_Username" L1977
formvalue 1 "ctl00_mnuLeft1_mSignin_mnuSignin_Password" larshg
submit

code 200     # make sure form submission is correct

save_html kandu3.html

I can't really figure out how to activate the login