The Minnesota-based Internet provider U.S. Internet Corp. has a business unit called Securence, which specializes in providing filtered, secure email services to businesses, educational institutions and government agencies worldwide. But until it was notified last week, U.S. Internet was publishing more than a decade’s worth of its internal email — and that of thousands of Securence clients — in plain text out on the Internet and just a click away for anyone with a Web browser.

Headquartered in Minnetonka, Minn., U.S. Internet is a regional ISP that provides fiber and wireless Internet service. The ISP’s Securence division bills itself “a leading provider of email filtering and management software that includes email protection and security services for small business, enterprise, educational and government institutions worldwide.”

Roughly a week ago, KrebsOnSecurity was contacted by Hold Security, a Milwaukee-based cybersecurity firm. Hold Security founder Alex Holden said his researchers had unearthed a public link to a U.S. Internet email server listing more than 6,500 domain names, each with its own clickable link.

Drilling down into those individual domain links revealed inboxes for each employee or user of these exposed websites. Some of the exposed emails dated back to 2008; others were as recent as the present day.

Securence counts among its customers dozens of state and local governments, including: nc.gov — the official website of North Carolina; stillwatermn.gov, the website for the city of Stillwater, Minn.; and cityoffrederickmd.gov, the website for the government of Frederick, Md.

Incredibly, included in this giant index of U.S. Internet customer emails were the internal messages for every current and former employee of U.S. Internet and its subsidiary USI Wireless. Since that index also included the messages of U.S. Internet’s CEO Travis Carter, KrebsOnSecurity forwarded one of Mr. Carter’s own recent emails to him, along with a request to understand how exactly the company managed to screw things up so spectacularly.

Within minutes of that notification, U.S. Internet pulled all of the published inboxes offline. Mr. Carter responded and said his team was investigating how it happened. In the same breath, the CEO asked if KrebsOnSecurity does security consulting for hire (I do not).

[Author’s note: Perhaps Mr. Carter was frantically casting about for any expertise he could find in a tough moment. But I found the request personally offensive, because I couldn’t shake the notion that maybe the company was hoping it could buy my silence.]

Earlier this week, Mr. Carter replied with a highly technical explanation that ultimately did little to explain why or how so many internal and customer inboxes were published in plain text on the Internet.

“The feedback from my team was a issue with the Ansible playbook that controls the Nginx configuration for our IMAP servers,” Carter said, noting that this incorrect configuration was put in place by a former employee and never caught. U.S. Internet has not shared how long these messages were exposed.

“The rest of the platform and other backend services are being audited to verify the Ansible playbooks are correct,” Carter said.

Holden said he also discovered that hackers have been abusing a Securence link scrubbing and anti-spam service called Url-Shield to create links that look benign but instead redirect visitors to hacked and malicious websites.

“The bad guys modify the malicious link reporting into redirects to their own malicious sites,” Holden said. “That’s how the bad guys drive traffic to their sites and increase search engine rankings.”

For example, clicking the Securence link shown in the screenshot directly above leads one to a website that tries to trick visitors into allowing site notifications by couching the request as a CAPTCHA request designed to separate humans from bots. After approving the deceptive CAPTCHA/notification request, the link forwards the visitor to a Russian internationalized domain name (рпроаг[.]рф).

U.S. Internet has not responded to questions about how long it has been exposing all of its internal and customer emails, or when the errant configuration changes were made. The company also still has not disclosed the incident on its website. The last press release on the site dates back to March 2020.

KrebsOnSecurity has been writing about data breaches for nearly two decades, but this one easily takes the cake in terms of the level of incompetence needed to make such a huge mistake unnoticed. I’m not sure what the proper response from authorities or regulators should be to this incident, but it’s clear that U.S. Internet should not be allowed to manage anyone’s email unless and until it can demonstrate more transparency, and prove that it has radically revamped its security.