CIFAR AI Frontiers School 2026

Exploring diverse perspectives of AI safety

After two decades of the Deep Learning + Reinforcement Learning (DLRL) Summer School, the Canadian Institute for Advanced Research (CIFAR) started a new format this year, the CIFAR AI Frontiers School. The first such event with around 30 participants was held in Toronto and dedicated to the topic of AI safety. It was a privilege to learn from experts and I am grateful for the opportunity. This blog post summarizes my highlights and personal take-aways of the three days.

Social sciences and humanities in AI Safety

Prof. Catherine Régis, Co-Director of CIFAR’s Canadian AI Safety Institute (CAISI), gave an overview of how CAISI aims to surface the best research to allow policy makers to make better decisions. Because most topics of AI safety are of socio-technical nature, rather than only technical, involving social sciences and humanities (SSH) is crucial. In particular, AI alignment is so difficult because AI systems touch all layers of society and many effects cannot be measured. A key question is how humans can supervise and oversee AI systems effectively. Some fundamental tensions in the development and regulation of AI are:

• Innovation and regulation. Recent years have seen a surge in AI research and deployment efforts, while regulation is lagging behind. The legal system, which traditionally provides a reliable framework based on rigorous and trusted norms, is currently overwhelmed with the fast-changing landscape of AI.
• Self-regulation and binding regulation. While companies and organizations have some incentives to self-regulate their deployment of AI, binding regulation from policy makers such as nation-states is likely necessary to provide a safe framework for AI to operate in. An example of successful binding regulation is browser cookies: Both companies and individuals can choose how to work within well-defined constraints.
• Venues for international agreement. Small forums that are relatively aligned in values, such as the G7, can agree quickly on targeted policies. However, because AI has consequences on human rights in every country, agreements among the United Nations (UN) are more powerful tools in the long term.

Indigenous perspectives

For this workshop, we split into different groups to learn and discuss about the intersection of indigenous communities and AI. My group was run by Anjali Mazumder, Director of AI Research at CIFAR. She opened the conversation by saying that papers that take AI safety seriously (e.g., when submitting to the FAccT conference, happening right after this summer school), authors usually need to provide three kind of statements: A positionality statement, ethics statement and generative AI statement. I had not heard of the positionality statement before, yet it made immediate sense: Authors are asked to reflect on their identity and background, which is important context when reading a paper that aims to deal with accountability and fairness. I also learned a handy acronym from another student: A system is SAFE when it is secure, aligned, fair, and explainable.

We discussed a concrete scenario, where an indigenous community is faced with changing migration patterns of caribou herds due to a new hydroelectric dam. It was a fascinating discussion where we brainstormed how AI experts can work together with the indigenous community on building a model to address this problem. While there were many more aspects and nuances to the discussion, two key questions were:

1. What side-effects and consequences does such a collaboration have? If we ask for location data for example, could this be detrimental if it falls into the wrong hands (e.g., non-indigenous illegal hunters) and does it pose an unwanted surveillance risk to the community itself?
2. What is the goal? When a community that is deeply in touch with nature comes together with modern technologies, what are we really trying to achieve? Just document the changed migration pattern and filing restitution claims in court, or is it more about predicting the new patterns? If it is the latter, does the indigenous community even want such technology and risk future generations becoming over-reliant?

Fairness by design

Prof. Laleh Seyyed-Kalantari from York University presented different perspectives of fairness in AI systems. She first provided unfairness examples such as underdiagnosis bias, occupational bias, and dialect bias. Whenever human data is involved in the training data, we risk building a system that is unfair toward a group of people in one way or another. To build fair systems by design, there are two distinct paths:

1. Fairness via awareness, identifying features that would cause unwanted bias and treating them differently than harmless features. Examples are individual fairness via counterfactual fairness (e.g., what would the system predict for a data point if we changed only the gender feature?), group fairness, demographic parity, and equality of odds/opportunity.
2. Fairness via unawareness, identifying and removing features that would cause unwanted bias. To measure the success of such an intervention, we can ask if the prediction of the model is the same when a certain feature is included or not in the model training.

I asked a question at the end, the answer to which helped me appreciate the difficulty of AI fairness: “How can we expect a model to not be unfair (e.g., sexist) when we give it features based on which humans have made unfair decisions in the past? Shouldn’t we just exclude such features (fairness via unawareness) and then there is no problem?” It turns out that even if we remove features that are obviously subject to human bias, there are often still traces of them left in other features. For example, AI models can differentiate between gender purely on x-ray images. Additionally, simply removing biased features risks unfair treatment of underrepresented groups, because now we can’t even apply fairness-via-awareness techniques, leaving minorities as outliers. Finally, excluding potentially harmful features from a dataset prevents analysis of their impact during training.

AI safety as a social and political construct

Prof. Geoffrey Rockwell from the University of Alberta began his talk with an important distinction between retrospective safety (e.g., having a “n days without an accident” in a workplace) and prospective safety (practices, systems, precautions). Both are important, but arguably the latter is particularly crucial for a sustainable future with AI. The most prominent framework to ensure safety is risk management, consisting of three stages: identification of harmful outcomes, assessment of their probability, and mitigation by implementing counter measures. However, risk management also comes with shortcomings: It is a technology itself (technosolutionism), hides the political, turns a blind eye to what is not measurable, and it only works for known unknowns, not unknown unknowns. Additionally, while AI safety is complex in itself, it is only a subset of the efforts needed for ethical AI.

The issue of safety has been around for a long time in all contexts of society and industry. Most of the fundamental best practices still hold for modern AI, but we also face a new challenge now: alignment. Here the question is what values should AI have to interact safely with humans and well-established assumptions? Different cultures have unique and sometimes conflicting viewpoints, which is why some companies adjust their recipe or design based on the country they sell their product in; same label, different execution. Perhaps in a similar way, AI-based products and processes should look differently depending on where they are deployed.

Can and should we even regulate AI? Technology is inherently dual use; a hammer can be used as a weapon or to build a house. However, comparing a hammer with AI is like comparing an orange to an apple tree, because AI is a much more broadly applicable and flexible tool, the uses of which are just beginning to understand. Thus, it is indeed crucial to think through consequences, intended and unintended. Prospective safety is required, we can’t just wait until a group of people decides to combine AI and CRISPR technology to build a harmful virus. Additionally, while safety regulations might be inconvenient obstacles for companies in the short run, they can also pave the way to progress. For example, the airline industry only took off after rigorous regulation.

As for the future of AI, it is true that there is unprecedented hype and investment in AI technology across countries and domains. However, the recent obsession with AI may not prove sustainable and the rhetoric of companies and governments will likely cool off eventually, leading to what could be described as an “AI fall.” It is however unlikely that we experience another AI winter as was the case a few decades ago, because modern AI already proves useful in many domains.

Technical frontiers of AI safety

Prof. Nicolas Papernot from the University of Toronto presented concrete examples of what modern AI can do, risks we already face, and counter-measures. When discussing AI safety, there are some things to keep in mind: Avoid anthropomorphising AI (starting with attributing “hallucinations” to language models), critically evaluate reports of emergent behavior (especially because models are often trained on data similar to the benchmarks they are evaluated on), and understand that outputs of generative AI can be extremely biased, even across different publishers. Furthermore, understanding the limits of the technology itself is not enough; because AI is deployed in socio-technical systems, we need to be aware of major challenges such as:

• Safety cannot be evaluated at the level of the AI component alone. For example, a harmful prompt can be broken down into a series of harmless prompts. That is why we need to think about AI safety not only on model level, but on (socio-technical) system level.
• Safety evolves as AI models are being deployed.
• Different stakeholders disagree on what safety is.

While the training and evaluation of AI models is messy, a sensible first step toward transparency was raised: Model makers should provide a binary oracle to regulators for any piece of information on the internet, indicating if it was used in training or not. This is technically feasible, does not leak too much information, and is useful to the public and policy makers trying to understand why a system behaves the way it does.

Panel discussion on the AI governance landscape

I enjoyed the fast-paced deep dive into the regulation side of AI safety by Jaxson Khan (CEO of Aperture AI), Teresa Scassa (University of Ottawa), Samir Chhabra (Innovation, Science and Economic Development Canada), and Foutse Khomh (Polytechnique Montréal), moderated by Elissa Strome (Executive Director of the Pan-Canadian AI Strategy at CIFAR). The speakers mostly agreed with each other, bringing their own expertise into the conversation. My main take-aways:

• Non-transparency is a big barrier to adoption. It also amplifies the low public trust of Canadians in AI.
• In the spirit of Canada’s AI for All initiative, recent legislation was proposed, such as bills C34 for safer social media and C36 for protecting consumer privacy.
• To test models safely, better technical sandbox protocols are needed. This could allow a decoupling between model training and evaluation, where a company or organization can test themselves if an AI model complies with their values and requirements.
• High-quality AI education is crucial, especially for the generations of students growing up with AI. The ability to critically judge the output of a system will become increasingly important.

Safeguarding courts from synthetic content

There was a fun interactive challenge, with the scenario to detect synthetic evidence submitted to a court case. Concretely, given a series of images, we should judge if they were authentic or manipulated using generative AI. The images were mostly real-world photos and scanned documents. This project is part of the CIFAR Solution Network program and the session was run by Kelly McConvey, Sajad Ebrahimi, Nima Jamali, and Maksym Taranukhin.

I ended up with a final accuracy of 63% classifying 30 images. This says a lot about the world we live in: It is really difficult to tell real from fake. We also had access to a an AI model, which we could query for its assessment as well as an explanation and a confidence score. The model was often uncertain, raising the important question whose judgement is more reliable: AI or human. Following up on this interactive session, Prof. Ebrahim Bagheri from the University of Toronto gave an overview of technical challenges regarding detection of manipulated evidence.

AI safety and democracy

Prof. Zhijing Jin from the University of Toronto gave an excellent overview of how AI poses risks to democratic and social systems. She highlighted a clear asymmetry: While generating AI content is fast and easy, it is expensive to fix. My main take-away was the nuanced presentation of different AI-driven threats to democracy:

1. Belief homogenization
2. Belief reinforcement
3. Congested bureaucracy
4. Epistemic flood
5. Unauditable authority
6. Normative centralization
7. Power concentration

Fireside chat with Nick Frosst

The CIFAR AI Frontiers School ended with a conversation between Nick Frosst (Co-Founder of Cohere) and Prof. Sheila McIlraith from the University of Toronto. Cohere helps companies deploy sovereign language models for their specific requirements and constraints. This is a narrower use case than jack-of-all-trades AI models designed for the general public, coming with some particular differences:

• Alignment is less of an issue within enterprise context, where the goal is to have a system follow its instructions within the context of company regulations. Instead of baking values and norms into a model, they should be included in the preamble.
• Companies usually have no intention to jailbreak a model; they want it to be useful in their context. While having strict authentication protocols in place is crucial to avoid misuse of a model, the outputs of the model itself usually require less filters than public-facing models.

Closing thoughts

I thoroughly enjoyed engaging with experts and peers in these three days, mapping out together what it means to develop safe AI systems. Some of my key learnings are:

• Not every problem can or should be solved with AI.
• Effective regulation requires compliance. Good policies are only effective if enforced.
• More open AI models increase the risk of bottom-up misuse; more tightly controlled models increase the risk of top-down misuse.
• My personal take-away is that I want to gain the technical expertise to build systems that follow AI policies and regulations, rather than drafting the regulations themselves.