Artificial Intelligence: Rule Based Machine Learning
It’s become almost impossible not to encounter or hear breaking news about Artificial Intelligence on a daily basis. ChatGPT, facial recognition, and autonomous vehicles all use a technology called Artificial Neural Networks. The first ANN was implemented back in 1958 (see Perceptron), but it wasn’t until 2013 that Andrew Ng, cofounder and head of Google Brain, could proclaim “we now have fast enough computers and enough data to actually train large neural networks”.
We may already be experiencing the golden age of neural networks, but they currently have two major drawbacks:
- Creating them is very expensive computationally, which means a large amount of real money paid for cloud-compute resources.
- The prediction model created by the network is a completely opaque black box.
The practical implication of a black box model is that the creators of the network have no way to understand how it works. How inputs are transformed into results is inaccessible to a human. There are ways to tune a model after undesired outputs occur, but no way to validate beforehand what outputs will occur. This can even lead to unpredictable results called artificial hallucinations.
You may have heard the recent news of a law team using ChatGPT to help prepare a brief that cited six imaginary cases provided by the chatbot. The prosecuting lawyer was called out on this and had to explain to the judge that his legal team was not aware that the chatbot could provide false information.
As Eric Clapton says:
It’s in the way that you use it
It comes and it goes
It’s in the way that you use it
Boy don’t you know
Ronald Regan would certainly be unhappy that we’ve turned his favorite proverb on its head, regressing* to: verify everything first, then trust.
* The statistical pun is also intended, I realize it’s bad form to explain your own joke…
Current neural networks can do amazing things, but some questions can also be answered by another AI technology called Rule Induction. A prediction model built using rule induction can be less expensive to create. But perhaps its most valuable feature is that it produces a collection of rules that can be clearly understood by a data scientist. To be fair, things like chatbots and automated vehicle navigation cannot be accomplished using rule induction, but it can accurately solve many classification problems, and the human understandable model it creates can be very useful for data mining.
In 2000, I wrote a topical research paper at the Illinois Institute of Technology (available here) about a relatively new but promising branch of artificial intelligence called called rule induction, which was being actively researched and incrementally enhanced over several years of cooperation within worldwide Machine Learning circles. And I recently discovered that the AMiner online service had picked up my paper at some point, probably because of its bibliography. “AMiner (funded by the National Science Foundation of China) is designed to search and perform data mining operations against academic publications on the Internet, using social network analysis to identify connections between researchers, conferences, and publications” [Wikipedia].
I decided I ought to reread my paper to see how well it held up after more than 20 years. The specific topic is an improved rule induction algorithm named Repeated Incremental Pruning to Produce Error Reduction (RIPPER) published by William W. Cohen at AT&T Bell Laboratories in 1995. After rereading both my paper and Cohen’s I did some internet research and was surprised to find that not only is RIPPER still being taught and discussed in academic and AI circles, it is currently available for use in Java, R, and Python projects.
- The popular Java-based open-source software Weka includes it
- The popular proprietary RapidMiner software includes a derivative of it
- The combination of open-source packages caret and RWeka make it available to R programs
- An open-source Python library named wittgenstein became available in 2019
- A research paper about it was included in the 2016 IEEE International Conference on Computer and Information Technology
- A 2019 lecture about it in a University of South Florida graduate Data Mining course
- On the popular Medium site, a 2020 article
- On the popular GeeksForGeeks site, a 2020 article
To wrap up, neural networks are currently very popular and very powerful, but the model they produce is inaccessible to humans. An alternative for some categorization problems is rule induction which produces a human-readable model. The open-source RIPPER rule induction algorithm, published in 1995, is still being used, taught, and discussed. It’s obviously stood the test of time, and if you work on AI projects, you might consider it the next time you reach into your toolbox.
