This is the second in a Build in Public series about how Passbolt uses generative artificial intelligence and how AI is changing the cybersecurity and open source ecosystem in general. The first in the series was about AI and cyber security, you can read it here.
As a cybersecurity company, there are concrete ways that AI has changed the ecosystem, making things more challenging for companies and individuals working in software, and especially those in cybersecurity. But the ethical dilemmas around the training and everyday use of generative AI are even wider-reaching, since nearly all white-collar workers have some interaction with AI tools. The ethical questions are diverse: How do you handle hiring? How do you evaluate vendors and open source projects, if AI is very good at creating the appearance of security and reliability, but you need to ensure both for your customers? How is AI impacting the mental health of programmers, and does that count as an ethical issue?
Tough Training
Let’s get it out of the way: There are ethical issues with how many LLMs are trained. They were given access to information that they shouldn't have been given and disregarded copyright laws. People mostly choose not to think about this. “It’s like when you buy a steak in the supermarket,” said Diego Lendoiro, Head of SRE at Passbolt. “You don’t think about the process to get the steak. You just want to feed yourself. And in a world where productivity is king, you don’t think too much about ethical challenges.”
So how do you find the equivalent of free-range, grass fed LLMs? One way is to move away from US-based proprietary LLMs to open source, open weight models that are hosted in Luxembourg, where Passbolt is based — and this is something Passbolt is doing. Luxembourg’s government offers companies located in the country access to open-weight models deployed on a government-owned supercomputer. “It makes it really easy to spin up sovereign models that are hosted nationally, and also protected by the government’s security teams,” said Kevin Muller, CEO and co-founder of Passbolt.
There’s also the environmental problems with AI. Data centers eat up water and electricity, and there have been ample examples of data centers causing both water and electricity shortages for the communities they’re located in. AI with less environmental impact isn’t something Passbolt can address directly, but it’s something the team is certainly aware of.
Asymmetry
In the previous Build in Public article I talked about how the cybersecurity challenge when it comes to AI is all about asymmetry: It’s an extremely useful tool for mounting attacks and finding vulnerabilities, but it is not very good at fixing the vulnerabilities. That asymmetry means vulnerabilities found at the speed of an LLM but fixed on a human timeline.
LLMs have made it very easy to produce code, to produce text and to produce slick-looking slide decks, websites, and even products. “In order to understand the quality of the production is actually there, it takes an insane amount of time, especially for cybersecurity products,” Muller said, about the challenge with evaluating these AI-driven products.
“For me, it is not ethical to ship things that you don’t fully control,” Muller said. “It’s like you enter a restaurant and the chef has zero clue of what he’s putting on your plate.”
To extend the analogy, the way generative AI is used in a lot of contexts shifts the responsibility for what is on your plate onto the diner. On the other hand, as long as the ‘chef’ is able to explain clearly when and where AI has been used, what data it has touched, and is clearly in control of the process from beginning to end, there is no ethical problem.
This clearly presents practical problems, not just ethical problems. How do you choose which vendors to work with, which open source projects to rely on? How do investors choose which startups to fund when they are sent seemingly identical pitches that all seem too perfect to be true?
This actually puts those trying to evaluate the security of a product, or the viability of a startup, in a bind. When the team at Passbolt decides which vendors to work with, they try to break through the slick surface with third-party certifications and audits that can’t be faked by an algorithm, as well as transparency so that they can verify for themselves whether or not a product or project meets security criteria.
Human resources, human relationships
One of the (many) places that the asymmetry shows up is in the hiring process. “We see a lot of CVs that are clearly an AI prompt,” said Arianna Ramirez Teran, HR specialist at Passbolt. She said she’s seen the same people applying for different jobs at Passbolt, and each time their CV was magically perfect.
Ramirez Teran admits that, from the job seekers view, they probably assume that there is an AI tool being used to scan their CV, and they’re just trying to get past those first filters. But she’s also genuinely looking for people who are actually interested in Passbolt, rather than those who are mass spamming thousands of companies at the same time. And she manually reviews all the CVs she receives, and does a video call with all of the good candidates. Again, the ease with which people can create a perfect-sounding CV, combined with the effort to discover whether or not the fit is as perfect as it sounds, just creates more work for the recipient.
Indeed, if you think of an application for employment as the beginning of a relationship between humans, it falls into a clear no-go area for using AI. “Where we don’t want to use too much AI is in human relationships,” Muller said. “You do not use your AI agent to talk to me on Slack. I am a human, you are a human, please talk to me in your own terms.” And if you haven’t read and understood every word in a document or message, you shouldn’t ask anyone else to read it until you do.
Policies on AI use
The explicit policies in place at Passbolt regarding AI use are more about security and data protection than ethics. “We are not allowed to use it (AI) when the code or the repository we are managing contains secrets, to avoid the risk of brute forcing them,” Lendoiro said. They also can not put any personal information about employees or customers into LLMs, with one exception — Passbolt permits limited use of Gemini through Google Cloud Platform, under its enterprise agreement and the associated data protection terms. This applies only to data and uses covered by existing agreements.
There is, however, a catch, one that all companies should be honest with themselves about. As with many other types of policies, you can forbid your employees from doing something, but it is impossible to be completely certain that no employee ever puts customer or employee data in an unauthorised LLM; there’s no alarm mechanism if it happens.
In fact, this push and pull between being cautious about data privacy and data security and using AI as a productivity tool is something that Lendoiro has thought about, too. “Once you start using it (AI), it slowly creeps in every single process and every area,” he said. “If you want to be productive with it, you have to fully commit. And as a cybersecurity company, that is probably not a good idea, because as I said, we can not use it in repositories that have secrets.” This is an under-discussed potential problem with policies that push employees to maximize their AI use; it’s very easy to imagine that they start using AI even in situations where it is clearly not a good idea for data privacy and security.
Maintaining the craft
What about policies that we’ve seen at other companies that force people to use AI? Passbolt doesn’t micromanage the way that employees use AI one way or the other, except to create policies around data privacy and security; as well as the unofficial guidelines not to use AI for human to human communications. But Lendoiro sees another ethical problem with companies who force employees to rely on AI: It can make them lose their touch.
“The more you use AI, the more detached that you are from the system,” he said. This can mean that people’s skills atrophy, and it can also mean that when something goes seriously wrong with the system no one understands it well enough to fix it.
That leads to something that is more of a philosophical question for a lot of tech workers. Do the companies they work for still value their skills and their craftsmanship, and more importantly, does the ecosystem as a whole value them? The pride in building something when it’s been substantially done by AI is just not the same as the satisfaction from coding software by hand. Are there ethical issues with an ecosystem that has taken away a class of workers’ sense of satisfaction and pride in their work?
“In the past, when you’re building something, you're proud of what you built,” Lendoiro said. “Right now I don’t have that (when working with AI), and for me that’s tough. Because it was the whole point of why I joined the tech scene, and it’s taken away from me.”