Website requires logging in now

[TinyNinja]

Hey,

I need help with logging into a website now. I have been downloading a file online for awhile now and figured I could just code the action to save the time. Now this month the website has put the files behind a user account. I have a username but I'm not sure how to pass that info through the code to have it login and download the file.

I have used Chilkat to make the downloading easier and a few login examples with zero success. Does anyone know how to pass the correct username through Chilkat or have used a different method to get through the login part?

Any help would be very much appreciated!

loHttp = CREATEOBJECT('Chilkat_9_5_0.Http')

loHttp.BasicAuth = 1
loHttp.Login = "foo"
loHttp.Password = "bar"

lnSuccess = loHttp.Download( lcUrlLoc , lcFileSave )
IF (lnSuccess <> 1) THEN
    ? loHttp.LastErrorText
ENDIF

~~ Update ~~
I use to go to the spot on the website that had the files I needed and download them from there. I saw the URL that triggered the download and took that link and just used that to download the files I needed and called it a day. Now they restricted the website more and need to have an account to download the files. They have a website and a login spot and you put in your username and password and then go to where the files are located and just click on them to download and save them to where needed.

IDK what the code should be that would take care of the username and password part so I can go back to downloading with my code again.

 

[Joe Crescenzi]

Although it's not a best practice, there is a way to pass credentials to a URL by adding a prefix.

http://myuser:mypassword@example.com will connect to the destination with that username and password.

For example to pass the user Frank and Password FranksPassword to example.com, you can try:

http://Frank:FranksPassword@example.com/something.aspx

 

[mmerlinn]

Although it's not a best practice, there is a way to pass credentials to a URL by adding a prefix.

http://myuser:mypassword@example.com will connect to the destination with that username and password.

For example to pass the user Frank and Password FranksPassword to example.com, you can try:

http://Frank:FranksPassword@example.com/something.aspx

You have me confused. You say to add a prefix to the URL? So, if I add a prefix to the URL, wouldn't if follow the form PREFIX + URL?

For example, Frank:Frankspasswordhttp://example.com/blah.aspx

Your example shows splitting the URL and adding the credentials inside of the URL.

 

[Joe Crescenzi]

You have me confused. You say to add a prefix to the URL? So, if I add a prefix to the URL, wouldn't if follow the form PREFIX + URL?

For example, Frank:Frankspasswordhttp://example.com/blah.aspx

Your example shows splitting the URL and adding the credentials inside of the URL.

I know it looks odd, and isn't best practices, but it actually works. I've even used it for web services when no other option was available.

Just check that syntax... you left out the @

For example, http://Frank:Frankspassword@example.com/blah.aspx

 

[Joe Crescenzi]

I know it looks odd, and isn't best practices, but it actually works. I've even used it for web services when no other option was available.

Just check that syntax... you left out the @

For example, http://Frank:Frankspassword@example.com/blah.aspx

Also... you can test it by literally typing the complete URL with a real user id and password. Once you get that worked out, try it within your code.

 

[Joe Crescenzi]

Pardon my dyslexia... this is how that should've read like my original post:

https://Frank:FranksPassword@example.com/blah.aspx

 

[TinyNinja]

Hey,

I tried the URL trick you mentioned and it did not do anything. It just reset back to the normal URL.

Looks like I will need to have something different to solve the problem.

 

[Joe Crescenzi]

Hey,

I tried the URL trick you mentioned and it did not do anything. It just reset back to the normal URL.

Looks like I will need to have something different to solve the problem.

It depends on how they manage security.

This method only works if you have a Windows Server, and it's a Windows login and password that server recognizes, not a database of account logins and credentials such as a WordPress or other database of users. Essentially it bypasses the default anonymous account used for web traffic.

 

[TinyNinja]

I'm not use how they are hosting their stuff. Before I could just go to the spot on the website that had the files I need and download them from there. I saw the URL that triggered the download and took that link and just used that to download the files I needed and called it a day.
Now they restricted the website more and need to have an account to download the files. They have a website and a login spot and you put in your username and password and then go to where the files are located and just click on them to download and save them to where needed.

I want to add some code to my code that will include the username and password so I can go back to downloading with my code again.

 

[Joe Crescenzi]

I'm not use how they are hosting their stuff. Before I could just go to the spot on the website that had the files I need and download them from there. I saw the URL that triggered the download and took that link and just used that to download the files I needed and called it a day.
Now they restricted the website more and need to have an account to download the files. They have a website and a login spot and you put in your username and password and then go to where the files are located and just click on them to download and save them to where needed.

I want to add some code to my code that will include the username and password so I can go back to downloading with my code again.

That's what I suspected when you said it didn't work.

If website requires a login that's part of their app, not a system account, the best you can hope for is that they also offer a way to either pass a token or API key, or let you log in manually from and the site has a way to keep you signed in using a cookie that it recognizes... or if the URL changes to add a session ID at the top, you could copy that revised URL (with the session ID embedded). It would work as long as the session or cookie doesn't expire too often.

 

[TinyNinja]

Dang that is a bummer. I'm not sure if they will be doing those options in the future or not.

 

[Chris Miller]

The login is at a specific URL, isn't it? It's surely in the form of a HTML form that's submitted. You have to "emulate" that by code. That's not possible by the simple basic auth approach Joe suggested, that requires a HTTP request with a request body sent over, that includes the HTML form data usually sent by the browser.

'You have to look into the HTML of the login page to know whether the HTML form is submitted with a GET or POST action and a few more details, before you could do the necessary HTTP request emulating that manual login automatically. The target URL for the login will also come from the HTML form definition, so all in all it would be necessary to extract the login HTML form from the login page HTML.

 

[Chris Miller]

By the way, Jow, AFAIK even with https protocol on a secure tunnel the URL itself is cleartext in the transfer, thus also a basic auth pair of crediantials, username and password will be transferred in clear plain text. So you hopefully won't see any basic auth authentification "prefix" mechaniusm embedded in URLs anywhere anymore. If you spot that used somewhere, make them aware it's unsafe.

PS: not 100% sure of the whole URL being clear plain text, but I think the upset against surveilance breaking privacy, if not by government than by commercial mass surveillance is that no matter if you use the internets most common protocol http with or without a secure tunnel (http vs https) the URL is the part that always has to be cleartext to know what server you want to reach, it's impossible to encrypt something (the request) for a target server including the target servers URL thereby not tell where the request has to go to. Now, the userassword@ part may be taken aside and be encrypted, too, but I doubt that.

 

[TinyNinja]

I just grabbed an example of Chilkat codes that might be used with the HTTP support. They have several different examples and I have tried a few with zero luck.

I download the file from here and that's the path for the login.
https://archive.org/ & https://archive.org/account/login

Any examples of logging in and then grabbing the file with Chilkat or something else?

 

[Chris Miller]

The key is to have the HTML form and then emulate submitting that HTML form in th same way a browser does, It's about sending over your credentials within the request body to the HTML form target. concentrate on understainding what the login page does by understanding the HTML.

Trying to throw in username and password within the http request is not getting you there, that's done outside the HTML form and so, won't help, ever.

 

[Chris Miller]

Here's some information about doing that. You have to know a bit from the underlying HTML to know what to send aned in which format and with which request method:

Sending form data - Learn web development | MDN

As we'd alluded to above, sending form data is easy, but securing an application can be tricky. Just remember that a front-end developer is not the one who should define the security model of the data. It's possible to perform client-side form validation, but the server can't trust this...

developer.mozilla.org

And don't look for VFP sample code, but you can set http headers with a httprequest object, for the Chilkat control, see:

Visual FoxPro POST application/x-www-form-urlencoded Two Ways

Don't just try, know what you need from inspecting the site source code, as the MDN article shows, the HTML <form> tag will contain the action to do, i.e. to which URL to send the form data and the method used, which could be POST or GET, etc.

So this requires a bit of both HTML and HTTP knowledge.

 

[TinyNinja]

Hey Chris,

Thank you for this help, this is a great call out. I was thinking that might be the Chilkat example to look at but wasn't sure. I will play around with this to see what I can do to get it to work. I played with HTML years ago and know the basics.

 

[Chris Miller]

Don't mainly play with the Chilkat example, first find out about the lgin page HTML form tag.

What action is it doing?
That's the target URL

What method does it use?
GET/POST mainly - that determines whether form data is posted in the request body or added into the URL, For logins it should be POST, but you only know looking into the HTML form tag.

Then form data in a request body can be Content-Type: application/x-www-form-urlencoded or Content-Type: application/x-www-form-urlencoded or Content-Type: multipart/form-data and a few more. Without knowing the login web page HTML nobody can give you a code example, because the necessary details are missing. There are a lot of different possibilities your two crediantials username and password can be posted to the login script.

And then that's also just step 1. Which component you use and whether it establishes a session that continues with further requests decides, whether this login step actually makes the next request, the actual download GET request (most likely) work.