Tek-Tips is the largest IT community on the Internet today!
Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!
-
Congratulations IamaSherpa on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!
Data Mining vs. Terrorism 2
- Thread starter Predictor
- Start date
- Thread starter
- #2
Predictor wrote:
"Recent tragic events have made me wonder about the ability of data mining to combat terrorism and other forms of crime. Does anyone know about such applications?"
Seb (over on comp.ai.neural-nets on Usenet) answered:
"not at all...and i think that's not so strange, because if (and i doubt it) these techniques would be used, wouldn't it be better to keep it quiet?
What applications are you thinking of? Mining in airline passenger lists? What if the next attack will be targeted on, say, a subway? I think it's an illusion to think that there is a solution to this kind of terrorism.
But of course, it's nice to brainstorm on this topic."
I am not suggesting that data mining is a "cure" for terrorism, simply one more tool to apply to the problem. I have heard of (but have no hard details on) data mining being applied to criminal activities via association rule discovery in an attempt to find relationships among sets of people, such as money launderers. I agree that it's an interesting technical subject, and of course it would be for a worthwhile cause.
"Recent tragic events have made me wonder about the ability of data mining to combat terrorism and other forms of crime. Does anyone know about such applications?"
Seb (over on comp.ai.neural-nets on Usenet) answered:
"not at all...and i think that's not so strange, because if (and i doubt it) these techniques would be used, wouldn't it be better to keep it quiet?
What applications are you thinking of? Mining in airline passenger lists? What if the next attack will be targeted on, say, a subway? I think it's an illusion to think that there is a solution to this kind of terrorism.
But of course, it's nice to brainstorm on this topic."
I am not suggesting that data mining is a "cure" for terrorism, simply one more tool to apply to the problem. I have heard of (but have no hard details on) data mining being applied to criminal activities via association rule discovery in an attempt to find relationships among sets of people, such as money launderers. I agree that it's an interesting technical subject, and of course it would be for a worthwhile cause.
- Thread starter
- #3
Predictor wrote:
"Recent tragic events have made me wonder about the ability of data mining to combat terrorism and other forms of crime. Does anyone know about such applications?"
Warren S. Sarle (over on comp.ai.neural-nets on Usenet) responded:
"Yes. Methods such as cluster analysis and link analysis are becoming popular in both law enforcement agencies and private companies to detect and investigate crimes."
"Recent tragic events have made me wonder about the ability of data mining to combat terrorism and other forms of crime. Does anyone know about such applications?"
Warren S. Sarle (over on comp.ai.neural-nets on Usenet) responded:
"Yes. Methods such as cluster analysis and link analysis are becoming popular in both law enforcement agencies and private companies to detect and investigate crimes."
- Thread starter
- #4
Predictor wrote:
"Recent tragic events have made me wonder about the ability of data mining to combat terrorism and other forms of crime. Does anyone know about such applications?"
Warren Sarle (over on comp.ai.neural-nets on Usenet) answered:
"Yes. Methods such as cluster analysis and link analysis are becoming popular in both law enforcement agencies and private companies to detect and investigate crimes."
Tim Josling (over on comp.ai.neural-nets on Usenet) responded:
"It is very difficult to use data mining to detect extremely rare events. Look up the base rate fallacy on google for details.
To some extent you can get a good proxy for the rare events by looking for less severe forms. An example would be the NY zero tolerance policy. It is known that minor law breaking is correlated with major law breaking, so by looking got small violations you can get at the major law breakers too. But I am not sure what the a minor form of terrorism is."
I think you are focusing on direct prediction (a classification problem), but I think other data mining technologies, such as more of association rules, deviation detection or text mining might be more applicable.
"Recent tragic events have made me wonder about the ability of data mining to combat terrorism and other forms of crime. Does anyone know about such applications?"
Warren Sarle (over on comp.ai.neural-nets on Usenet) answered:
"Yes. Methods such as cluster analysis and link analysis are becoming popular in both law enforcement agencies and private companies to detect and investigate crimes."
Tim Josling (over on comp.ai.neural-nets on Usenet) responded:
"It is very difficult to use data mining to detect extremely rare events. Look up the base rate fallacy on google for details.
To some extent you can get a good proxy for the rare events by looking for less severe forms. An example would be the NY zero tolerance policy. It is known that minor law breaking is correlated with major law breaking, so by looking got small violations you can get at the major law breakers too. But I am not sure what the a minor form of terrorism is."
I think you are focusing on direct prediction (a classification problem), but I think other data mining technologies, such as more of association rules, deviation detection or text mining might be more applicable.
- Thread starter
- #5
Predictor wrote:
"Recent tragic events have made me wonder about the ability of data mining to combat terrorism and other forms of crime. Does anyone know about such applications?"
Warren Sarle (over on comp.ai.neural-nets on Usenet) answered:
"Yes. Methods such as cluster analysis and link analysis are becoming popular in both law enforcement agencies and private companies to detect and investigate crimes."
Tim Josling (over on comp.ai.neural-nets on Usenet) responded:
"It is very difficult to use data mining to detect extremely rare events. Look up the base rate fallacy on google for details.
To some extent you can get a good proxy for the rare events by looking for less severe forms. An example would be the NY zero tolerance policy. It is known that minor law breaking is correlated with major law breaking, so by looking got small violations you can get at the major law breakers too. But I am not sure what the a minor form of terrorism is."
I think you are focusing on direct prediction (a classification problem), but I think other data mining technologies, such as more of association rules, data visualization, deviation detection or text mining might be more applicable. Here are some interesting Web sites:
"Recent tragic events have made me wonder about the ability of data mining to combat terrorism and other forms of crime. Does anyone know about such applications?"
Warren Sarle (over on comp.ai.neural-nets on Usenet) answered:
"Yes. Methods such as cluster analysis and link analysis are becoming popular in both law enforcement agencies and private companies to detect and investigate crimes."
Tim Josling (over on comp.ai.neural-nets on Usenet) responded:
"It is very difficult to use data mining to detect extremely rare events. Look up the base rate fallacy on google for details.
To some extent you can get a good proxy for the rare events by looking for less severe forms. An example would be the NY zero tolerance policy. It is known that minor law breaking is correlated with major law breaking, so by looking got small violations you can get at the major law breakers too. But I am not sure what the a minor form of terrorism is."
I think you are focusing on direct prediction (a classification problem), but I think other data mining technologies, such as more of association rules, data visualization, deviation detection or text mining might be more applicable. Here are some interesting Web sites:
- Thread starter
- #6
Paul Victor Birke (over on comp.ai.neural-nets) wrote:
"I assume you are looking for an outlier in amongst some clusters.
This is of course very hard because likely the data would be noisy.
You have to sort things out though....
You would have to locate a "real outlier". This is difficult to prove.
See a discussion by Montgomery at ASU on Design of Experiments about outliers and their importance.)
You may have the outlier and maybe 10 real noisy points that look like outliers."
I was thinking that other data mining technologies (text mining, for instance) might also be used. I agree that the approach you describe (deviation detection) would not be definitive, due to the large proportion of false positives. Consider, though, that having 11 suspects (= 10 noisy points + 1 real terrorist) may be a significant reduction in caseload for further investigation.
"I assume you are looking for an outlier in amongst some clusters.
This is of course very hard because likely the data would be noisy.
You have to sort things out though....
You would have to locate a "real outlier". This is difficult to prove.
See a discussion by Montgomery at ASU on Design of Experiments about outliers and their importance.)
You may have the outlier and maybe 10 real noisy points that look like outliers."
I was thinking that other data mining technologies (text mining, for instance) might also be used. I agree that the approach you describe (deviation detection) would not be definitive, due to the large proportion of false positives. Consider, though, that having 11 suspects (= 10 noisy points + 1 real terrorist) may be a significant reduction in caseload for further investigation.
- Thread starter
- #7
Tom Osborn wrote:
"Exploratory and evidence assessment methods are being used pretty widely (without being reported much - and rightly so).
Much of it uses association methods (link analysis, and some kinds of bottom up clustering based on associations), and
confirmatory methods (to filter out low- from high- likelihood suspects). A lot of this might be called "the wrong place at the
wrong time, talking to the wrong person(s)" method.
This kind of work can't be done on its own. By identifying prospects and information gaps, the gaps and further data can be
sought deliberately to complete the picture."
That's very interesting. I realize that much of this is not published, but do you have a references to any of this sort of work,
perhaps online?
"Exploratory and evidence assessment methods are being used pretty widely (without being reported much - and rightly so).
Much of it uses association methods (link analysis, and some kinds of bottom up clustering based on associations), and
confirmatory methods (to filter out low- from high- likelihood suspects). A lot of this might be called "the wrong place at the
wrong time, talking to the wrong person(s)" method.
This kind of work can't be done on its own. By identifying prospects and information gaps, the gaps and further data can be
sought deliberately to complete the picture."
That's very interesting. I realize that much of this is not published, but do you have a references to any of this sort of work,
perhaps online?
- Thread starter
- #8
pwaksman (over on Yahoo's Data Mining Club) wrote:
"I have a vague suspicion that Americans will accept some cutting back on their civil liberties in order to assume greater protection from terrorism. So I expect some more discussion of phone-tapping and email monitoring. Data Mining will probably need to play a role in protecting the innocent (in addition to identifying the likely guilty) for that to be acceptable."
I imagine the data mining-related security measures which would most likely raise privacy / civil liberty concerns would be face recognition (which has been used in public locations already) and e-mail / phone monitoring.
pwaksman continues:
"Here is an "outlier" processing scenario: form a small population of known (but rare) outliers cases. Build up statistics for this small ("guilty"
population. Also build up statistics for the larger ("innocent" population. Now in comparing the two sets of
statistics some patterns will be present in both; others will be unique, or more likely, in the small ("guilty" population. It is *these* patterns that are best for identifying a single "guilty" individual. Proper use of Data Mining ought to be able to tell
practisioners if they have found something significant or not. So it is a case of having good technology both help find the guilty and protect the innocent."
I wonder. I don't know that enough terrorist examples could be collected to make this work. I suppose that even if it worked weakly, it might help filter the data.
"I have a vague suspicion that Americans will accept some cutting back on their civil liberties in order to assume greater protection from terrorism. So I expect some more discussion of phone-tapping and email monitoring. Data Mining will probably need to play a role in protecting the innocent (in addition to identifying the likely guilty) for that to be acceptable."
I imagine the data mining-related security measures which would most likely raise privacy / civil liberty concerns would be face recognition (which has been used in public locations already) and e-mail / phone monitoring.
pwaksman continues:
"Here is an "outlier" processing scenario: form a small population of known (but rare) outliers cases. Build up statistics for this small ("guilty"
population. Also build up statistics for the larger ("innocent" population. Now in comparing the two sets of statistics some patterns will be present in both; others will be unique, or more likely, in the small ("guilty"
population. It is *these* patterns that are best for identifying a single "guilty" individual. Proper use of Data Mining ought to be able to tell practisioners if they have found something significant or not. So it is a case of having good technology both help find the guilty and protect the innocent."
I wonder. I don't know that enough terrorist examples could be collected to make this work. I suppose that even if it worked weakly, it might help filter the data.
-
1
- #9
I think that the Outlier theory could benefit from a comparison with a population of known criminals and their profiles, would be necessary thoug to be able to generate a map of the criminal organization and it's reach, also it is interesting to understand that the organization even then they try to seem ramdom they are actualy acting under an orgnized system that has to make sense so action can be sincronized even then they are plan by separet cels like an corporate enviroment that even though the HQ may not know what each cel is doing it still has the guide line for profit and what can be done and when.
I think war againt terrorism can benefit from datamining but I think that an effort to really know the criminal population would be better that just statisticaly try to predict since the hipothesis may not be applyed to all cases, therefore I think that profiling the organization and it's reach and implication is more effective. AL Almeida
NT/DB Admin
"May all those that come behind us, find us faithfull"
I think war againt terrorism can benefit from datamining but I think that an effort to really know the criminal population would be better that just statisticaly try to predict since the hipothesis may not be applyed to all cases, therefore I think that profiling the organization and it's reach and implication is more effective. AL Almeida
NT/DB Admin
"May all those that come behind us, find us faithfull"
-
1
- #10
jeanpierre
Technical User
Data Mining is already used in an embedded manner to tack for terrorists. This can be through :
Text Mining - that scans e-mails, faxs for risks and set template. All e-mails and faxes in/out bound for the Untied Kingdom are intercepted and scanned. I am positive the same exists for USA and other NATO countries. The proble in scanning all this info is, of course, the quantity of info it generates. The amount simple swamps the agencies involved. Further passes of analysis are conducted on "interesting" data but it still ultimately depends on human intepretation - and this requires large manpower,
Voice recognition - is used to scan thousands of voice communications simultaneously. Again the technology is there - can be better but the ultimate problems lie with what the human do with the info and how swamped they are.
In my humble opinion, there is a need to use multiple avenues to dispose of terrorist. Other skill the US has not placed at its full strength is propaganda and good-will aimed at countries that potentially react adveresely to the USA. USA was very successful at containing the communist country because there was an understanding of the commies culture and a multitude of services were aimed at these countries to defuse tension and build revolt among the populace.
Text Mining - that scans e-mails, faxs for risks and set template. All e-mails and faxes in/out bound for the Untied Kingdom are intercepted and scanned. I am positive the same exists for USA and other NATO countries. The proble in scanning all this info is, of course, the quantity of info it generates. The amount simple swamps the agencies involved. Further passes of analysis are conducted on "interesting" data but it still ultimately depends on human intepretation - and this requires large manpower,
Voice recognition - is used to scan thousands of voice communications simultaneously. Again the technology is there - can be better but the ultimate problems lie with what the human do with the info and how swamped they are.
In my humble opinion, there is a need to use multiple avenues to dispose of terrorist. Other skill the US has not placed at its full strength is propaganda and good-will aimed at countries that potentially react adveresely to the USA. USA was very successful at containing the communist country because there was an understanding of the commies culture and a multitude of services were aimed at these countries to defuse tension and build revolt among the populace.
- Thread starter
- #11
Someone else mentioned examining earlier, less severe legal violations, and I heard on the news recently that some of the hijackers had immigrations violations before the attack. Even if this didn't identify all terrorists, it might lead to the identification of many more by association with those with known legal violations.
The problem with the immigration violations are that the law obligates people to ilegal even when they want to be legal, basicaly there are three ways for an ilegal alien to become legal, if I'm not mistaken:
1) Be a 'Slave' Migrante Worker on plantation field or a very specialized worked sponsored by a Company that can prove could not find an american citizen to the position,
2) Running away from a communist Country or sufering Politic Procecution at origin country,
3) Married to an American Citizen
Any other reazon to stay in the country longer than 3months are considered ilegal. AL Almeida
NT/DB Admin
"May all those that come behind us, find us faithfull"
1) Be a 'Slave' Migrante Worker on plantation field or a very specialized worked sponsored by a Company that can prove could not find an american citizen to the position,
2) Running away from a communist Country or sufering Politic Procecution at origin country,
3) Married to an American Citizen
Any other reazon to stay in the country longer than 3months are considered ilegal. AL Almeida
NT/DB Admin
"May all those that come behind us, find us faithfull"
- Status
Similar threads
- Locked
- Question
- Locked
- Question
- Locked
- Question