Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

  • Congratulations Chris Miller on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!

Browsers/blockers stopping http_referer 2

Status
Not open for further replies.

1DMF

Programmer
Jan 18, 2005
8,795
GB
Can any one tell me if there is software or browsers or indeed toolbars or plugins which would deliberately withold the HTTP_REFERER variable.

Or if there is a setting to turn this option on or off.

thanks
1DMF
 
Hey Foam, there was no need for a rant, it's a matter of perspective and personal preference. I don't agree with anyone who thinks they need to hide HTTP_REFERER, is there something your trying to hide ??

I couldn't care who knows what websites I have visited, i'm not sure how they think this information is going to be of any use to them.

they may know what site i visited, but if they use this information for any reason whatsoever they are breaching the Data Protection Act 1984/1998, it's not knowledge/information that is dangerous, it's what you do with it.

as for cookies, like you said cookies can be turned off, and people get the choice, so you agree using cookies that might stop people from using their site.

from a commercial perspective it's obviously not desirable to put restrictions like this on something you want to sell to prospective clients!

so that applies to cookies also...

for some reason you don't see the irony of the P2P software that a user installs and gives every other million people running the software access to a complete area of their computers hard drive - being a privacy thing ????

why beacuse these "reputable" [bigsmile] shareware software providers wouldn't put a back door in some where for themselves or even indeed hackers who might find or make one.

Oh just like the writers of Quake never put a back door in - yeah right! - now do you see my point!

The problem is because of the way the project is meant to work, it's not the visitor to the clients site my program will be discriminating against or affecting, it's the clients using my software.

which is why for me it is such a dilema.

in my experience the majority of "Joe Public" , don't care as long as things work, i see this every day the frustration people get trying to work out virus software, popup blockers, firewalls, SP2 etc....

Witholding the last web page they visited I assure you isn't even on their list of concerns , infact the thought whether they should or shouldn't withold this information hasn't even crossed their mind, or even concidered the possibility of if this information can be obtained or with held.

They are simply mr & mrs John Smith of no 32, just trying to fill in an application form request to recieve a sales pack regarding joining our network.

computer's you gotta love em, just you wait, if I ever meet Sir Clive Sinclair , I've got a few choice words to say to him.









 
Hey, I've got an idea: Build a working time machine, go back to when they were developing the http protocol and convince them to make it compulsory to return an HTTP_REFERER.

OR adapt your code to fit the way the world is, rather than how you think it should be.

Your choice.

-- Chris Hunt
Webmaster & Tragedian
Extra Connections Ltd
 
That's a corking idea Chris, now I just gotta remembered where I parked the Tardis.

But if i find i've been clamped, any ideas on how you adapt code to do what i require via a static HTML submitted form.

is there any other environment, or other variable that I might be able to authenticate some how with?
 
I have come to love how people who would readily call themselves "professionals" to clientel are unable to discuss and resolve an issue without regressing to the mentality of a 4 year old child.

Any web application that puts a dependancy on an optional HTTP request header is, as some point, going to fail and making such a big issue about it, is frankly embaressing to watch.

Almost every browser sents a HTTP referer field unless specifically told not to or the referring page has no URI (i.e it was typed in by hand). I would say 99% of Internet users don't even know about the existence of the HTTP referer field, what it does or have any clue how to disable it. As several people have pointed out, its seems that you assume that it is the user who has taken the steps to hide the value when in fact third-party software and hardware are often the culprit.

Its very easy for the URL of a interal intranet page or other "secret" URL to accidentaly be transmitted if it was the last page a user visited before going somewhere else.
Many organisations purposely block or mangle the referer header because of this and you'll often find values like xxxx:++++++++... as the referrer where this has happened. This is either done via a proxy or masked out in some hardware firewalls.

As Foamcow mentioned, home Internet Security software such as Norton also does this and, thanks to the history of security holes in your beloved Internet Explorer, is becoming more prevelant and desired by end users.

The HTTP referer field is an optional field for information purposes only. It was never intended to be used as part of an access control or authentication system and by its very nature is too unreliable to be used as such. It it also too easily abused and a lot of spyware and malware will purposely change the referer to the URL of a site they wish to poision your logs with.

It is very easy for it to be changed by anyone with a little skill. Hence, should someone discover that any referral from "xyz.com" be enough to access your application, they would only need to force that domain into the header to gain access - a trivial task.

It seems to me that what you want is "silent" authentication so that there is no requirement from the user to have to conciously assert their right to use your form and associated scripts and for that there are many much simpler, secure and much easier to maintain solutions.

Foamcow has already suggested sending an "ident" as part of the form data to the script which can be checked. The only flaw with this is that the originating site needs to included this code in the HTML and you will NEVER be able to hide this in plain text from anyone visiting your site.

Another big issue is that the client (i.e the machine the visitor is access the site from) will be variable and will not always have a constant I.P address. However, the web server where the pages are being server from will (in 99.99% of cases).

A smarter solution would be to proxy everything via the remote web server itself. The form "action" submits the form data to a local server script. This script then forwards all the data to the remote script on your server. Because this data is being forwarded to you by the server, the IP/Host it originates from will be constant and therefore much easier to authenticate. Of course, being a proxy, when your remote script provides a response the local script will need to echo the response back to the client.

Proxies like this are trivial and very easiliy written in languages like PHP, ASP, Perl, etc. and can easily be included as part of the "package".

Of course, there are a lot simpler ways of solving this but it would require the client to authenticate themselves using a traditional name/password system, but again wanting to use HTTP_REFERER originally shows you don't really want to go down that route.

Anyway, apart from all this, thanks for the amusement your banal ramblings about W3C standards being "wrong" for IE, especially the one about the pointer. Why you need to set it anyway is beyond me as the "hand" cursor is the default cursor for any link, be it text or image. Trying to force it to be something will probably be what breaks it, rather than fixes it. Of course, you could of just used style="cursor: pointer;cursor: hand" and saved yourself the anguish and obvious sleepless nights.
 
for some reason you don't see the irony of the P2P software that a user installs and gives every other million people running the software access to a complete area of their computers hard drive - being a privacy thing ????

er.. no.
In that case they made a concious decision to install the P2P software and use it. You seem to forget that bit.
They may also have elected not to share any files anyway.
If the P2P software opens up parts of the machine that the user has not granted permission to do so then THAT is a privacy breach.

You are confusing "access" with "privacy".

Reference to backdoors is out of place since if they did they would be breaching the privacy act you mentioned.
AFAIK production of P2P software is legal, use for downloading copyrighted material is not.

The problem is because of the way the project is meant to work, it's not the visitor to the clients site my program will be discriminating against or affecting, it's the clients using my software.

Except that the visitors are possibly using browsers that won't pass the information your script relies on to provide the service to your clients.


I do think your idea is a reasonably good one and I hope you can come up with some authentication method that will work.
I'll keep trying to think of an answer. Wunderboy is on the right track in my opinion.

One more question.
How do you ensure the mail that is delivered (presumably HTML) renders properly in your clients mail app? Or do you create PDF's/Jpegs of the form?



Foamcow Heavy Industries - Web design and ranting
Target Marketing Communications - Advertising, Direct Marketing and Public Relations
I wonder what possesses people to make those animated gifs. Do you just get up in the morning and think, "You know what web design r
 
Wunderboy, oh how i love it when people miss the point completely.

I never said the user was witholding anything, I made a comment about the irony of people who do and then use P2P software.

the point about the cursor is about how people who make rules aren't necesarily right, as its a pointer turning to hand, which once again the point is completely lost on most reading this along with you thinking all images and links automatically do this?.
Code:
<img src="blah blah" onclick="window.open(blah blah)">

is there something wrong with my browser because mine doesn't change to hand when the image is hovered over?

unfortunately irony is lost on most people, It makes no odds to me how or what is blocking it, it's the fact it is, which makes you question why the environment variable exisits if it's of no use to man nor beast.

once again the simplicity of the system i have written is lost on those reading, and what i am trying to achieve.

all i asked was is there another way of authenticating JUST LIKE http_referer but isn't blocked and is acurate.

Instead of everyone getting insensed over the rights and wrongs of witholding it, you could have simply said.

NO THERE ISN'T!



 
FoamCow, please i understand people install P2P willingly, and beleive me there are probably back doors, i gave a prime example of Quake to justify this.

what i'm saying is on the one hand they deliberately block something simple as the HTTP_REFERER but willingly install suspect software.

like my last post its an irony thing thats all, most of my comments are meant to be taken tongue in cheek, but I think people aren't getting that.

I'm gald you do appreciate what i'm trying to achieve, i have a fully working version , which needs rewriting now that those lovely guys like ishnid, stevexff, kevinADC, PaulTEG etc.. have taught me many things over the last few months, down to correct use of workspace variables and writing my own PERL modules, so it needs a complete rewrite since my new improved skills have been obtained, but there is no point in me bothering if I can't get round this simple problem.

unless I accept the system could be open to abuse and sacrifice security just to enable greater compatibility with everones privacy settings.

The IP address of the server with the form on it is the last resort, but I do not know the environment variable needed to ask for the server's IP rather than $ENV{'REMOTE_ADDR'}.

can you help?

I do lock down the user interface so not only do you need a username and password, but only a specified IP can log on to create, edit, delete the templates in use.





 
FoamCow , I forgot to answer your last question, this only works for those using HTML email systems, either client or web based.

but as it's my clients generally that recieve the email and they are corporate running some kind of exchange with outlook or equivelant it isn't an issue.

I've also found that even if it was to be an interactive email that the client would send to their email base, if the recipient doesn't have HTML compatible email then they aren't interested if that recipient cant view or reply.

seings as most people should have HTML compatible email this isn't really an issue, although i am aware before another scuffle breaks out over this comment that some choose to set their email to text only. well that's your choice and good luck to you.

Personally, I'm all for richer, higher quality, better use of modern technology, but then again some people aren't like me as if you hadn't noticed :p

a good example of compromise would be this,
I DJ on an internet radio, that some might say discriminates against modem users and low BroadBand capabilites, since they upped the quality of the stream you can't broadcast unless you have a fairly decent internet connection, but on the other hand, the listener now gets a better quality of broadcast.

You can't please all the people all the time, or make anything 100% compatible all the time, sometimes a compromise has to be made.



 
It's not the discrimination that is the issue really.

It's the statements that aren't quite "fact".

but as it's my clients generally that recieve the email and they are corporate running some kind of exchange with outlook or equivelant it isn't an issue.

It sure is. There is no guarantee that it will work. An email server is just 1 cog in the whole thing. For example, I use a PC with Outlook and a Mac running Entourage to connect to our Exchange server. Both display HTML differently.

You'd think that with our Exchange server that receiving the mail would be no problem? Wrong. A spam filter is killing it and stopping the mail getting through. I might add that the HTML is basic and nothing gets flagged up by content checking software at our mailing house.

seings as most people should have HTML compatible email this isn't really an issue, although i am aware before another scuffle breaks out over this comment that some choose to set their email to text only. well that's your choice and good luck to you.

The problem comes when a mail client "thinks" it's HTML compatible but isn't. Or when the mail client interprets the HTML in it's own special way.
I'm working on an HTML email campaign at the moment, it's alot of fun because my client uses Lotus Notes. Try sending an HTML email to someone using that. It's like AOL on a bad day.


I've also found that even if it was to be an interactive email that the client would send to their email base, if the recipient doesn't have HTML compatible email then they aren't interested if that recipient cant view or reply.

Then why send them an HTML email at all?
Or do you think people like mailboxes full of unreadable/useless mail?

Send multipart mail at the very least. Or properly target your recipients based on their preference.

I'll concede that there is nothing really wrong with HTML mail providing you accept it isn't going to work all the time (been here before haven't we!). If your users find it works for them, then great. You are dealing with a kind of closed target group so you should be ok.

I'd still look at the possibility of PDF generation though. I think that would improve your service even more.

Each to their own I guess. But if I were you, I'd spend some time looking at how things actually work rather than how you believe they should work. Please, no offence is intended.

Anyway all that's another issue isn't it.

Foamcow Heavy Industries - Web design and ranting
Target Marketing Communications - Advertising, Direct Marketing and Public Relations
I wonder what possesses people to make those animated gifs. Do you just get up in the morning and think, "You know what web design r
 
I was going to let this lie, but curiosity led me to read the same question in the PERL forum.
I feel I need to put something straight concerning "tetchiness".

Yes, I said I was tetchy and used Firefox. The 2 things are not necessarily related. It's a personality trait.

The reason FF users are tetchy is that they are constantly faced with ignorant opinions on what a browser should and shouldn't do. The simple fact is that FF, so far, has come closer to behaving like a browser is meant to than any other browser. Coupled with it's speed, size, ease of use, features and great extensions it makes a good choice for the savvy user to use on a day to day basis.

Now take a look at this forum, count the percentage of questions relating to "issues" with IE and compare that to the relative percentage market share. And the people here are IT literate. What does that say about how great IE is?
Stuff that really shouldn't be a problem is a problem.

As a developer you should be concerned with how your work appears in "other" browsers than IE. Because to be frank, IE is the buggy one and many people have learnt methods to get around these nuances that simply do not work well with standards compliant browsers. And you bet that standards are important. Look at VHS video, DVDs, Railways, Librabries, Kitchen cabinets and appliances.... what do they have in common? They all conform to set standards. Standardised things make life easy. Fiddly, proprietary things make life difficult. Fact.

It's possible to tell a site developed by someone like yourself, primarily for IE with a flagrant disregard for anything non IE, and one developed by someone that actually understands the web (not HTML, PHP, ASP or Javascript but the web itself).
The IE code is normally bloated, illogical and over engineered. It's also, more than likely, "broken" in some way and probably uses or even relies on IE proprietary tags or Javascript. But hey, it "works in IE and everything else sux".
That's sad really.

I really hope you solve your authentication problem. I also really hope that you open your eyes a bit and realise that the web has moved on a bit in the last 10 years and doesn't, despite their best efforts, actually belong to MS.

Foamcow Heavy Industries - Web design and ranting
Target Marketing Communications - Advertising, Direct Marketing and Public Relations
I wonder what possesses people to make those animated gifs. Do you just get up in the morning and think, "You know what web design r
 
Excellent retoric FoamCow, you hold a good argument, I tried to give you a star but then realised I'd already given you one so I hope someone else does on my behalf.

I have to make the point that some of this has nothing to do with the right or wrongs but more to do with frustration.

I don't agree with all you have said, and i'm certainly not a devout MS follower, in my experience and this is FACT, I have never, ever, had a problem with IE not working whatsoever!

yet other browsers I have, FACT!, so you cannot say one is better than the other due to compliance, I like one better than the other because it works!

I do not design especially for IE, I have other sites which run fine in FF, which I have installed and use for W3C testing of my site along with W3C checkers for CSS validity, although i won't do it for HTML.

if you think because it is compliant and standard to have to end tags such as <b> <font><u> etc.. before a </td> command yet in I.E. it does it for you with the </td> sorry I know which browser is more intelegent, programmer friendly and nicer to work with and it certainly isn't W3C compliant.

it's healthier and better and more environmentally friendy to cycle to work, but i still drive , why coz it's quicker and easier. you might call it lazy, i call it smart.

you see it's a different in not just opinion but also philosophy and outlook on life.

And if the web comminity worried less about if an end tag command was used, the internet would be a nicer friendlier place to be.

and why did your spam filter block the email, it wasn't spam if you filled in the form and submitted it to yourself, not more usless software, when will it end!



 
And if the web comminity worried less about if an end tag command was used, the internet would be a nicer friendlier place to be.

With a statement like that, I pity anybody who has the misfortunate of needing to visit any web site you have developed.

Standards are there for a reason. You're right - you don't have to follow them. But don't expect blatent anti-following to gain any kind of sympathy from developers who understand why they are in place.

Dan


[tt]Dan's Page [blue]@[/blue] Code Couch
[/tt]
 
I didn't ask for sympathy, and once again you miss the point, broswers handle missed end tag commands w3c calls it non compliant.

but people like you would expect the browsers to not work for code that wasn't 100% compliant, don't supose you were in the army by any chance?

Respect to all browser developers that continue to develope out side of the box, respect and love to you all!
 
You'd make a good sergeant major, believe me.
 
The spam filter caught MY mail. I was using it as an example of how you can't rely on stuff working. I've not tried sending your form through it.

Our spam filter is horrendous It's nothing to do with me though, so I can't change it. I wonder how much other mail we don't get :)

I think you need to rethink WHY your code sometimes works in IE and not in other browsers.

Surely it's better to tell the browser what you want it to do rather than relying on the browser manufacturer to guess what you want to do? That practice can lead to unpredictable results, which is why your pages sometimes have problems in other browsers.

Each browser renders in it's own way at the whim of the browser creator. If you leave it to the browser to decide what you actually mean with your code then of course your page won't work as you wish in some circumstances.

That's what standards are about. They are not some draconian measure to bash MS or make you conform. They are about giving you control and, above all, confidence that what you spend time creating works as you want it to.

So in regards to this statement:

if you think because it is compliant and standard to have to end tags such as <b> <font><u> etc.. before a </td> command yet in I.E. it does it for you with the </td> sorry I know which browser is more intelegent, programmer friendly and nicer to work with and it certainly isn't W3C compliant.

It's nothing to do with intelligence. Other browsers will also still display the page if you don't close tags. You just can't guarantee how they will do it.

Simply because the code may display as you wish in IE does not mean it's "right". It's just IE's interpretation of the markup.

What W3C standards do is they make sure that the interpretation of markup is reasonably consistent across the board.

Now, it's impossible to expect all the browser manufacturers to magically decide things should be done the same way if they have to account for every possible coding nuance all us developers used to do. That's not practical.
But by creating a set of frameworks for how markup should be structured the W3C have enabled the browsers to know what you want it to do and render the content accordingly.

Standards are in their early days really, so not all browsers have caught up yet. But the manufacturers think it's a good idea and so it's all going that way. If you don't embrace the idea you are going to come unstuck I assure you.

but people like you would expect the browsers to not work for code that wasn't 100% compliant, don't supose you were in the army by any chance?

No, that's not correct either.
We expect the code to work in the way the browser manufacturer decides it should. That is a problem since it is more than likely not not the way it's intended to work.

Take a step back. Stop thinking standards are "out to get you" or make your life harder. It's actaully quite the opposite. You know the best bit? It's actually really easy to work to set standards than it is to patch, fix, sniff and worry about how your site works in other browsers on other platforms.

Foamcow Heavy Industries - Web design and ranting
Target Marketing Communications - Advertising, Direct Marketing and Public Relations
I wonder what possesses people to make those animated gifs. Do you just get up in the morning and think, "You know what web design r
 
lol - don't take it personally FoamCow, eveyone's out to get me!

I genuinly take on board all your comments, and of course i want it to work in as many browsers as possible, and when I get the chance, many of my sites will be written to be more compliant.

Alot was more not to do with me refusing to accept W3C, but legacy code from being self taught before I even knew there was a standard, or that half these issue would bite me, as the internet matured.

Unfortunately, just being compliant today, won't make you compliant proof, as standards will change, along with the technology.

I'm still anti-flash, even though my computer and Broadband, now gives me less of an excuse to say they are SLOW!

I still find flash annoying, distracting, and more often than not will close a site as soon as it opens if it is in flash, especially if they have an into page i can't skip! or music i can't turn off.

I mean where would we be without our gif's to give us moving picures :p

 
Unfortunately, just being compliant today, won't make you compliant proof, as standards will change, along with the technology.

Perhaps, but because the standards were documented the browser of the future will know how to deal with our "old" sites.

See the benefit now?

Foamcow Heavy Industries - Web design and ranting
Target Marketing Communications - Advertising, Direct Marketing and Public Relations
I wonder what possesses people to make those animated gifs. Do you just get up in the morning and think, "You know what web design r
 
Come on FoamCow, even you know some legacy standards are dropped , just because something is documented on how it worked, doesn't mean the next generation will care or make it backward compatible.

I see errors in code all the time with the line
"That command has been deprecated".

And more importantly it's the industry that wants things to have a life cycle, it's all about money!
 
Status
Not open for further replies.

Part and Inventory Search

Sponsor

Back
Top