Errors in machine learning algorithms are creating critical (and nearly invisible) consequences
(This article first appeared in Towards Data Science)
Do you use a mobile banking app?
We have almost certainly paid thousands, if not millions, of dollars for returned checks that aren’t actually bad.
I’m not sure if anyone is aware of it.
It is impossible that I’m the only person that this has happened to.
But it’s easy to see how we might all be missing it.
The technology behind mobile banking is pretty incredible, but what happens when there’s a mistake?
What happens if we don’t see the mistake?
We’re living in a world where so many technological advancements have been made that they almost blend into the background. We’ve gotten used to the idea that we can let our phones and computers do the little things for us. It’s easy to forget how new all of this technology really is.
But it is new. It’s changing every day. There are algorithms behind most of the basic things that you take for granted, from social media and entertainment to banking and finances. They are constantly evolving.
They are not perfect.
Pay attention!
If an image capture system makes an error within your banking app that causes your deposit to be rejected, what will that cost you?
What if no one sees it? What will that cost us all?
This is not a bad check!
Recently, I deposited a check on a mobile banking app that was accepted, only to be returned as a bad check a few days later. I was charged a fee for this.
Here’s the problem: that check was not bad.
Here’s the bigger problem: this took almost two weeks to sort out and involved a huge amount of wasted time. It was practically by chance that I even caught the actual issue. There was an error in the image capturing system within a mobile banking app. An easy-to-miss mistake that may have already happened to you without you even being aware of it.
There are a lot of perks to working as a freelancer, but the financial side can get complicated. Rather than a steady stream of checks from a single source, you’re depending on a variety of clients to send you payments from different accounts on what can be a pretty random schedule.
I’m lucky enough to have amazing clients who I trust completely. Not that mistakes can’t happen! They absolutely can. People lose track of their balances, grab checks from the wrong account or a closed account, and so on. But I’ve been incredibly fortunate to have never yet had a client send a bad check.
You can imagine my surprise when, days after having a mobile deposit approved through the banking app for a major bank, that check was returned and my account was hit with a sizable fee. This is a client I’ve known and trusted for years, and it didn’t make sense. I reached out to the client, and he was as baffled by the situation as I was.
I contacted the bank and spent a very long time on the phone with someone who assured me that there was nothing that she could do. It was a bad check. When someone writes you a bad check, you have to pay a fee. You’re expected to have your client reimburse you for that fee. End of story. Your bank can’t possibly go to some other bank and demand that they pay the fee, so it’s up to you. Just go back to your client and get a new check.
Sounds legit, right?
Bad news, banker
Unfortunately for her, I wasn’t going to let this go. Here’s where I have the advantage over a lot of other people out there who might take what she’s saying at face value. People who would simply return to the client and request a new check. One with added charges to cover the fee and possibly interest as well:
- I know and trust my client.
- I was holding the check in my hand and could see that it was drawn on an account from this same bank. (This suggested that some information that she had was incorrect.)
- I am familiar with some of the algorithms driving image capture and classification as well as their potential weak spots. (I actually write about technology and artificial intelligence here on Medium.)
- I’m aware that this bank recently did a major tech upgrade.
I used to be a personal banker with this same bank. It was a long time ago, but I know what information a banker has access to and what steps she can and can’t take.
It’s unlikely that you are in this position and that’s why I’m writing this story.
How many people do think have simply gone back after depositing a check on a banking app, gotten a new check, and paid the fee without identifying the actual problem?
You will believe your banker. You will believe that you received a bad check and proceed from there. Your client will need to provide another check with additional fees. It could affect your relationship with them. The returned check may cause you to overdraw your account, incurring more fees and much larger problems. Multiple returned checks in a short period can cause you to lose your account. A lot of things can go very badly here, all because of an error in a machine learning algorithm.
I want you to have this information. I want you to know what you’re looking for. You can and should ask questions. Is the problem really with the banking app and not with the check? There are a lot of things a banker can’t tell you but plenty of information is available to you. Was the check returned because of insufficient funds? Is this an account that doesn’t exist? What other steps can you take?
What is really going on here?
Ask the questions!
So what happened?
That was surprisingly hard to figure out. I’m sure the banker on the phone hopes she never hears from me again. Eventually, it turned out that the account that the check was drawn on was not the account number entered into the system. That means that the bank couldn’t locate the bank account in question.
I had to wait for the bank to send me a certified copy of my deposit so that I could take that and the original check to a bank for a banker to examine in person. At that point, the banker could try to figure out the problem and determine whether or not to reverse the fee. (For added fun, your fee can generally only be reversed by the bank where you opened your account. That means that I’d have to wait for a banker I’ve never spoken with halfway across the country to decide whether to reverse my fee.)
So, I waited for the mail.
It took close to a week. It’s not hard to imagine what this would be like for someone who was now overdrawn on their account because of a mistake entirely on the bank’s end.
Take it to the bank
Once the check arrived and my toddler could be spared from the world’s most boring adventure, I headed over to the bank and went through all of this again in person.
This banker said once again that this was a bad account. The customer definitely existed, but the account wasn’t an open account. It must be one that had been closed and my client had grabbed some old checks by mistake. I’d just have to get a new check.
Easy mistake. It could happen to anyone.
This was all so plausible, but I know my client. I also know routing numbers. This guy was not the sort of person who would be holding on to a box of checks from a closed account that he opened in a state where he lived seven years ago. Not happening.
Somehow, even though the banker knew with absolute certainty that she was in the right, she looked again.
She saw the problem. It was so simple.
The image capture system that the banking app uses had cut off the last two digits of the account number on the check.
That’s it.
That mistake, both tiny and massive, caused a returned check, a fee, and nearly two week’s worth of headaches and wasted time that could have been productive.
It could have been far worse.
That bank error was practically invisible.
Enter artificial intelligence
So where does this image capture system come from? Is it unique to this bank?
It turns out that most of the major banks all use the same company for the image capture, recognition, and analysis systems within their banking apps. This company does incredible work and I’m in no way questioning that. I have zero proof that they are directly the cause of this issue.
The company uses artificial intelligence to develop algorithms for image recognition. They’re using machine learning algorithms to do incredible things with document and ID verification. They’ve created an image capture software development system built on computer vision and machine learning algorithms. It detects corners and glare, can detect and analyze images on a variety of backgrounds, contains built-in analytics, offers real-time image assessment, and has a lot of other cool features.
But it’s not, apparently, flawless.
That said, I don’t believe that this company is directly the cause of the problem. I know that the bank I use for my freelance work has recently undergone a major tech redesign and they’ve made changes to their online and mobile banking app. It might be possible that the redesign on the bank’s end caused a problem with the image capture system.
It also might be possible that there is a problem with the image capture system itself. I’m having a hard time locating any records of errors on the tech company’s part, but that doesn’t necessarily mean that there aren’t any. They might have an amazing PR department or a stellar legal team. Or there might not be a lot of other people who happen to be in a position to notice exactly what happened here.
I have no way of knowing where the fault lies or how often this happens. But it’s impossible that this has only happened one time.
This is potentially a massive problem. Even if 0.1% of customers who use mobile banking apps are having (or will have) this issue, that’s a huge problem within our economy. An enormous number of people use mobile banking apps and that number is growing. The company that makes this technology also builds technology that’s used in ID and document scanning. Can it just drop numbers? These are the numbers that represent our bank accounts and our identities. This kind of mistake is not acceptable and will have extremely serious ramifications.
What’s the solution?
I don’t know yet, but I know we need one. We definitely need to start training bankers to watch for this problem. Banking apps are not perfect. Remember that all of this technology is still new. Remember that you have the right to ask questions. You have the right to get to the bottom of the situation, even when people tell you that you are already there. This is almost certainly happening everywhere and we need to find a way to fix this problem.
It’s up to all of us to pay attention. No one is going to solve this problem if they don’t know about it.
Don’t let this slide. It’s too important.
If anyone else has had the same issue, feel free to discuss it in the comments below. As always, reach out any time on LinkedIn @annebonnerdata.
