I wanna create this movement. I wanna bring people together. I want to leverage all of the knowledge that's out there in so many different domains and learn from other people, and have other people learn from each other, because that's really what we need to get the solutions at the speed and the scale required in order to tackle such a complex problem like climate change. Good morning, good afternoon, or good evening, wherever you are in the world. Welcome to episode 222 of the Climate Confident Podcast, the go-to show for best practices in climate emission reductions and removals. I'm your host, Tom Raftery, and if you haven't already, be sure to follow this podcast in your podcast app of choice so you never miss an episode. Before we get going, a huge thank you to this podcast's, incredible supporters, Jerry Sweeney, Andreas Werner, Steven Carroll, and Roger Arnold. Folks, your backing helps me keep this podcast going and I truly appreciate each and every one of you. If you'd like to join this community, you can support the show for as little as three euros or dollars a month, which is less than the cost of a cup of coffee. Just click the support link in the show notes of this or any episode or visit tiny url.com/climate pod. Now you know that sinking feeling when a company drops a shiny net zero pledge and you just know it's greenwashed nonsense. Today's guest built an AI that calls BS on that. While ChatGPT is out there hallucinating climate facts, she built ChatNetZero and ChatNDC AI tools, trained specifically to fact check climate pledges, expose vague targets and make sense of the mess that is national climate plans. She's not waiting around for governments or big tech to get it right. She's making the tools we actually need to separate the serious actors from the smoke and mirrors. Her name's Professor Angel Hsu. She runs the Data Driven EnviroLab and she's here to explain why climate needs better data, smarter ai, and fewer buzzwords. But before we get into that, in the coming weeks, I'll be speaking with Sangita Waldron, who's the author of What Will Your Legacy Be?, Ciaran Flanagan, Global Head of Siemens Data Center Solutions. Stuart Thompson, President of ABB Electrification Services and Frank Maguire, VP Insight, strategy, and Sustainability for Sharethrough. Now back to today's episode, and as I said, my special guest today is Angel. Angel, or welcome to the podcast. Would you like to introduce yourself? Thank you so much, Tom. It's really a pleasure to be here. I'm Angel Hsu and I'm an associate professor of Public Policy and Environment, Energy and Ecology at the University of North Carolina at Chapel Hill. And I also founded and direct a research group called The Data-Driven EnviroLab. And our mission is to innovate quantitative approaches to evaluating the world's most pressing environmental challenges from climate change, to air quality, urbanization, and other related issues. Okay. And why, as in, what made you decide to go down this path? What was it that kind of clicked in your head one day and said, I think I want to get into dot, dot, dot? Well, if only it was that easy that these ideas just appeared to me. But actually when I was working on my dissertation in the late early two thousands. So I started my PhD in 2008. I recognised that in reading all of the literature on environmental management and strategy and policy, that there was this disconnect. And so, even though the business community and the private sector had for many decades recognised the value of data and evidence to guide business decisions. So I think we've all heard of the word KPIs or key performance indicators, and the idea that you have to get returns on your investments or ROIs. That, that same kind of logic of we need to actually collect data and track progress on these key efforts that we're taking to address environmental issues. That same logic and thinking had not translated into the environmental domain, and so for a lot of really key issues, you think about some of the most pressing environmental concerns. Poor air quality, lack of water resources, loss of tree cover and forest, and loss of biodiversity in species and rising temperatures. We still didn't have a lot of data to measure what is the extent of that problem, and then importantly, if we're implementing efforts to then tackle them. How are they actually performing and how do we connect the two? And so that's really where I started in my PhD. And I was lucky enough to actually spend some time in China. So I was actually living in Beijing right before the Olympics, and I saw how much of a issue air pollution was, and water scarcity and climate change was about to be a really big issue. They had just launched their first national climate strategy in 2007 and recognising that at that time they were also really interested in having data and tools to help them better manage their growing climate problem and their growing air pollution issues as well. And so I was working for an environmental think tank called the World Resources Institute and specifically on a project called the Greenhouse Gas Protocol Initiative. And so I was working with the Chinese government and Chinese businesses to help them utilise these tools to get just better baseline data because they were trying to manage a growing energy consumption problem, energy insecurity. And then also recognising that the source of that energy, which was fossil fuels, predominantly in the form of coal, was generating air pollution and also causing their greenhouse gas emissions to rise. So it was kind of the confluence of all these different issues where the Chinese were thinking at the time, we need to have better data and tool to understand how these various issues relate to each other and to guide our policies to determine what we should do and what kinds of investments we should make. And so after that experience, I left China. And I started my PhD recognising that we need to actually think about like how do we actually quantify and better develop database solutions to some of these pressing issues. So then I was studying data science and statistics, geography, looking at the role of remote sensing and satellite information to help fill in some data gaps. And then more importantly, to understand how stakeholders and companies, and also policy makers could then utilise the information and utilise the data to make better informed decisions, because it's one thing to have a lot of data, but then how do you actually translate that into the decision making context? So that's how I started to think about what this revolution could be and what it might mean for the world to be able to have better access and better information when it comes to key indicators for the environment. And talk to me about the data driven EnviroLab. How did that come about? And what is its aims and and functions? Yeah, so as I was finishing up my PhD, and it was really, as I mentioned, trying to merge together and leverage these different disciplines, data science and then geography and policy together in order to really solve climate change. I mean, that was like really my raison d'etre of wanting to, to work on environment and and climate and to get a PhD and to get this additional training was really to try to do everything that I can to tackle climate change. Acknowledging the fact that not a single discipline can tackle the climate issue alone. You can't just have earth scientists or you can't just have lawyers or social scientists in the room. Economists or political scientists. You really need to have everybody working together. And one of my strengths that I realised in doing my PhD was that I'm a connector. So I really, I, I like to learn from different people and I, I feel like in order to solve some of these really pressing and urgent environmental and climate issues, we have to have a sense of humility and recognising, again, that not one discipline, not one perspective, not one person can solve the issue alone. So that's when I started to go to my friends and colleagues who were trained in different disciplines, or maybe they knew about different domains like fisheries or forests or air quality, and I really took a lot of pleasure in learning from them and being able to bring them into the fold and to think about interdisciplinary solutions with data at the core. So one of the first projects that I started to work on as I was transitioning from my PhD into my research into my academic faculty position was the Environmental Performance Index, and that is a biannual index that evaluates 180 countries around the world on 20 key environmental performance indicators from environmental health, to the health of fisheries, to the role of an impact of agricultural systems on environmental sustainability. And that's really where I started to get the idea that. Well, if I can try to bring together people under this brand or this idea that we need to make environmental decision making more data based, more evidence driven, then that can really start to, to draw a movement and also help policy makers and, and decision makers and businesses realise that they should be leveraging all of these advances in data in order to better inform what they're doing in their own practices. And so, that was really how the Data-driven EnviroLab came about. And, and the fact that, like my last name, which is Hsu it, it's like, sounds a little funny. And growing up here in the American South, a lot of people didn't know how to pronounce it because it's spelled HSU. So I got a lot of, oh, is that Miss Hezoo, Zhou? You know, like a lot you could you think of, you name it, it's probably how people mispronounce my last name. And so I thought, you know what, I actually. I wanna create this movement. I wanna bring people together. I want to leverage all of the knowledge that's out there in so many different domains and learn from other people, and have other people learn from each other, because that's really what we need to get the solutions at the speed and the scale required in order to tackle such a complex problem like climate change. Okay. And. You're using AI, which is one of the the big things that you've Yeah. You've, you've, you've, I mean, AI is no good without data Correct. Yeah. You're, you've got the Data Driven EnviroLab, so it's kind of a natural progression right from there to, to AI. So what is it that you're doing with AI? Yeah, exactly. I mean it's really interesting because in my data science class last week I started to introduce these policies, master students to machine learning. And I told 'em, I said, it's really funny because I'm gonna be teaching you some very basic techniques, some unsupervised machine learning techniques, which basically 10 years ago were just called non-linear, multi, variate regression techniques. And so it's really funny to kind of see these trends kind of take hold and to latch onto methods that frankly have been around for decades. So AI, machine learning, not anything new, but we've kind of rebranded them. And I think now what's really exciting is that because it's so pervasive in every walk of life and in society, that now we actually have a face to them. And so it is exactly as you said, it was a natural progression where I was really thinking about. Okay. We have data out there and some of it's useful, some of it's not, and some of it needs probably some type of transformation, some type of normalisation, some type of like I guess pivoting towards a particular goal in order for people to actually make use of it and to take action on that kind of information. And then I'm somebody that doesn't shy away from new things, and I think that that's like probably pretty unusual for a lot of academics who they were trained in one thing, they're an expert in one thing and they kind of keep chipping away at the same problem as they go along. But I think part of the strengths of the Data-driven EnviroLab is that we're not afraid of embracing some of these new trends. And I think in order to be relevant and to be able to just stay relevant, you have to be open to all of these new technologies and these innovations that come along. So, at the beginning of my career, it was really thinking about like third wave or like even fourth wave forms of data. How do we engage citizens who all have cell phones and who can have sensors and their own ways of collecting information, help to fill data gaps? And then thinking about like satellite remote sensing, which has been around since the seventies, but not necessarily in a policy and business decision making context. So how can we leverage all of this remote sensing satellite data that is by and large, free and publicly available into these kinds of contexts. And then of course, machine learning helps to recognise these patterns when you have like all of these different data sets. From sensor data or citizen volunteered information, you've got top-down data from satellites and other sources. How can we actually make sense of all these kinds of data and delve a more complete picture than the one that we may have had before, compared to like very traditional statistical data techniques that may involve like going out and like conducting surveys or like in-situ measurements or having like monitoring instruments or stations that are fixed. And so that's really, I think the natural progression of like getting into AI is like recognising that there's always the natural tendency and I think the desire to take a bunch of different data together, develop statistical models. You know what, what we do in the policy sense is we try to develop econometric or statistical models to try to make sense. If I've got like a y outcome and I've got all these different variables, how does X and all these predictor variables relate to Y and can X explain like what's going on? And why? mean, that's at the, at the base of what we're trying to do, but then the X can be very incomplete. And so it may only explain like a little part of Y. So then the idea is, well, how can we leverage techniques like machine learning, which simply don't look for linear relationships? I mean, they're really trying to look at non-linear and just really complex. You can think about like webs and different networks. So how can they leverage these multi-variate, multi-dimensional relationships within different variables in order to understand or explain the why or like the end goal of what we're trying to measure. So I think that's like, I don't know if that's like a really long-winded or confusing way to explain like why, you know, I've been interested in AI, but that. That to me is how my brain thinks about the logical progression of recognising, okay, we have all this data, but how do we actually bring it together? And AI? I think about it as like the glue and helping us like kind of tie everything together and make sense of it and help us, like yeah, better make sense of all of this. Yeah, the, I guess these like un unpredicted or sometimes surprising or like you don't necessarily come into a problem or like a bunch of data necessarily thinking, oh, well, all of these variables could have some sort of relationship to each other. And that, that's what AI helps us do. It helps us to discover the undiscoverable and then help to make the invisible visible. Okay. And obviously the AI revolution, it's not really a revolution, but the revolution that we've seen in the last two and a bit years since ChatGPT came to the fore has been kicked in the well, has been kicked off, shall we say, by ChatGPT, you've got a version of that, not ChatGPT, but ChatNDC. Tell us a little bit about that. Yeah, so I guess I didn't fully answer your, your last question, which is how we're using AI and so I, I named one more general application, which was like data fusion. So how can we take and, and leverage the fact that there's so much data now being collected by sensors and satellites and other methods by activity from our online behavior and preferences that we express through what websites we click on and what we buy. So there's a lot more data that's going on. So that's like one application. And I think this emerging application is also recognising that texts can be used as data as well, and, and can be quantified, can be analyzed. And that's really the foundation of these large language models that are looking at these patterns in text and then using that to develop these machine learning based models that can then help us to generate new text. So generative AI then can leverage these large language models and help develop these chat bots that make us think that we're actually communicating with an intelligent sentient being on the other side of end of the computer when we're really just interacting with a large language model. And that's really, I think where ChatGPT came online and really exploded the internet. I mean, it blew my mind when it first came out two years ago, and I started to ask a question. I think everybody, all these journalists, everybody was going on and asking it different questions and trying to test the limits of, of these types of chatbots and trying to figure out, okay, well if I can I try to get it to be in a relationship with me, or can I try to ask it to predict the future and all these kinds of situations. But then I started to ask ChatGPT about climate change because obviously that's my domain, that's what I'm interested in. So I started to ask it very basic questions like what does it mean to pledge net zero? Which of course is the overarching goal that society, the world climate scientists say that we have to strive towards to reach by mid-century if we have any hope of staving off the worst impacts of climate change. And so I asked it things like can we continue to use fossil fuels and have a credible net zero target. If you're a company and you've pledged to decarbonise, can you continue to use fossil fuels? And the, the answer that I got from ChatGPT was quite ChatGPT It said yes. A company that is pledging net zero isn't necessarily failing to make a credible net zero pledge. And that really took me aback because I'm a person who has contributed to these international assessments. The IPCC is probably the most well known for those listeners who are unfamiliar. That's the Intergovernmental Panel on Climate Change. That's the most authoritative scientific assessment of climate change science. It is now in its seventh, starting on its seventh assessment cycle. It started in the early 1990s. It's an intergovernmental process. So governments nominate scientists all around the world. And so that's really in its strength is that you have representation globally. It's not just a couple scientists. I mean, governments have the opportunity to nominate whoever they like. And then that process is, is then whittled down to several thousand scientists that then contribute in the end. And so what the IPCC says is that incontrovertibly, absolutely, you cannot continue to burn coal, natural gas and oil. And hope to achieve a credible net zero scenario in the timeframe that we need it. Like we really, all of the scenarios basically show a dramatic phasing out of fossil fuels. So when I saw that from ChatGPT, I was instantly worried and I thought, oh my gosh, like this is problematic. And then I started on this rabbit hole of like asking it a bunch of other climate policy related questions and realising that it does definitely does not have very accurate or up-to-date information. So if you were to ask it things like. Does Egypt have a net zero target? I mean, you can go and ask it now, and it will say yes. It will give you a series of policy documents. It'll give you certain years, and all of it is made up. And so this is the well-known problem with generative AI and a lot of these chat bots of hallucination. So it can generate very real, incredibly sounding answers that frankly are not grounded in any accuracy. And that's a well-known problem with a lot of generic chat bots because they're just trying to generate human and realistic sounding responses. But there's not necessarily verification that happens on the part of these large companies that are developing these large language models in chatbots. And so you as a user, you're none the wiser. You have no idea what's actually accurate and what's not, unless you actually then do your own research to try to figure out, okay, well is this actually true? Did Egypt actually pledge a net zero target? Can I continue to use fossil fuels and have a credible net zero target? But then again, who wants to read the tens of thousands of pages of IPCC report? Right? And so that's one of the reasons why my team and I have developed now two chatbots. One is ChatNetZero.AI, and that one is specifically targeted towards these net zero pledges and to try to make a a project the net zero tracker, which my lab is one of four principal groups developing. And so that project, the net zero tracker, we track the net zero pledges of 4,000 entities, including all national governments, and large regions in the G20 and then also cities over the size of half a million. And then all Forbes 2000 publicly listed companies and about a hundred or so privately listed companies are also included. And so we have a scorecard and we go and we search for information about whether or not these different entities have pledged a net zero target, and then we evaluate them for credibility. So have they pledged near term action? Do they have a transparent plan that details what they're doing? Do they include things like shady offsets that may not be focused on. And so their efforts are not focused on decarbonising their own efforts and their own supply chain and their own emissions. You know, things like that. And so even though we have a scorecard for every actor, it's really difficult to then very quickly say, well, I wanna know like, who's doing better? Is it Germany or the United States, or Amazon or Walmart? It's really difficult to do these kinds of comparisons and then to be able to ask like these questions of evaluation. I mean, sometimes it's hard, even if you're given data to then for you as a non-expert to make sense of it. So we thought that actually adding a layer of AI to just interact with the data that's on the net zero target, plus a few selected evaluation documents and these standards that the UN bodies have put together to specify what is actually a credible net zero target, that would be a much more constrained way allowing the AI to then only pull answers from verified documents and verified sources that we train it with, and then we've also implemented an anti hallucination algorithm to then make sure that every response that's generated is actually checked against one of our sources. And then we provide those references and page numbers. For the user to then verify themselves that the AI hasn't completely gone off the rails. So that's one of the, the chatbots. And then the second one you mentioned, which is ChatNDC.org. The NDC stands for Nationally Determined Contributions. So these are national pledges that countries have pledged to the Paris Agreement. But again, that's really difficult. A lot of them are not in English. They are incredibly variable in terms of their length and their content. And so if you just wanna ask a very simple question of does Egypt have a net zero target, or what has the UK pledged and how have their national climate policies changed over time? You can actually do that with ChatNDC, and so we're using AI, but we're not allowing the algorithm and the model to go search the internet and use everything available. Who knows. I mean, that's the problem with a lot of these generic models like ChatGPT and with Gemini, is that you really have no idea what they're using to train these models and how they're generating these responses. And so we've tackled that problem by trying to constrain what's available. So this is a process called retrieval augmented generation or RAG for those who are familiar with a lot of these AI terms. And so that's a much more constrained way to develop a domain specific chat bot that can produce much more credible and accurate responses that if you were to use a generic model. And who are these sites available to, and are they free or paid? And who do you think is going to be the main users of these sites? Yeah, so for right now, the tools are publicly available so anybody can use 'em. They're accessible to anyone. So I didn't mention that we also have a document catalog. So not only can you ask these two chat bots, just whatever question you want about, obviously like don't ask it like, what should I have for dinner? Or come up with a recipe for my like lunch today, because that's not what they're trained to do. Like continue to use ChatGPT for that. But if you're asking a climate domain specific question, then we also allow or we also have this document catalog feature where you say, okay, well I only want you to tell me information from like China's NDC or from like the IPCC documents. And then it will highlight where in the document it's finding the information from. So that same capability that we currently have in ChatNDC, we're actually adopting for ChatNetZero as well. So users will also be able to then ask like document specific questions. So then they can say, I wanna look at Apple's latest climate plan, what they're planning to do. And I want you to like actually highlight specific areas where it's talking about offset use or it's talking about their interim target or their net zero pledge. So those are some updates, but right now everything is available to the public and we're hoping that we'll be able to fundraise and scale some of these tools to allow like an a host of other really cool features. So some of the things that users have been asking as they say, well, I have my own set of documents that are proprietary or I don't wanna share on the internet. So can I upload those documents and then incorporate those into the model as well? And then be able to ask questions of, if I'm Apple, I'm picking on Apple, but we'll just say like, let's say I'm Apple and then I have like a draft of my new policy strategy, and I wanna say, okay, well I wanna know how this compares to my competitors. And that would include things like Samsung or like, you know, like, Intel or something, or these other type of Microsoft or these other types of companies. But I don't want that information to be available for the public. And so you these kinds of features as well. And then what we have heard from a lot of stakeholders and like policy users is a lot of people they wanna be able to do like forward-looking, like forecasting of like, what should I be doing, what could I be doing? Where are areas that I could be improving and if I were to enhance my climate policy efforts in these domains, is that going to help make the world better in terms of like reducing climate? You know, these kinds of, like, these are really complicated, you know, more fore casting questions. Whereas a lot of AI is much better at doing like the past, like based on a past set of data, historic trajectories and historic patterns. How can I make sense of what's been done? But I think this type of forecasting is much more challenging. So in terms of the users, I mean, I think already we're saying that a lot of different audiences are really interested and, and have started to use the tools. So I think on one level, individual users, so like, anyone who's like a researcher or a concerned citizen, if they just wanna get basic, credible information about climate change policy. I mean, I've even encouraged my students in my climate change class this semester as they're developing their policy memos. So they're doing a mock conference of parties, so a mock cop next week in class. And so one of the things I recommended that they do is to actually use my chat bots to try to get some information about like the NDCs in these different countries and what they've done, how they've evolved over time. So I think if you're a student or an educator, a concerned citizen and you just wanna get some basic accurate information, I think these tools are really useful. And then I think also if you are a policymaker. It can be really useful if you are trying to update any of your types of, or yeah policies. Even not a policymaker per se, but you're a decision maker and let's say that you're trying to inform a particular agency or constituent or a government agency about what they should do. I think this is also a really nice tool to get. Accurate information. And then I think also for like companies, it's, it's all really useful to help them benchmark. So if they say, I'm in the apparel sector and I really wanna know like what companies in the apparel sector are doing and where there are some gaps and like how I can be like leading in these given like a certain set of constraints. I think this is also like an area where these tools could be really useful. And we've heard from some like think tanks as well. We've actually been contacted by a couple of think tanks that have said, these tools can be really, really useful. Right now I'm manually sifting through country level NDCs to try to find information about climate finance and how they're tackling climate finance, how much they're pledging by way of climate finance to developing countries. Have they put any conditionalities on what they're pledging. That kind of information I have heard from different policy researchers and various NGOs and civil society groups that they're having to manually do that now. And they see a lot of potential in these AI driven tools. ChatNDC and ChatNetZero to help them streamline that process. And so, as another example for ChatNetZero right now, it's a very volunteer driven process where we actually have volunteers go through and they Google, they search for the different documents, whether it's like a country level policy document or a company level corporate social responsibility report. They have our framework for what we're looking for, and they're manually looking for that information. So in this next round, what we hope to do is to actually use ChatNetZero to help actually do that. Process that a volunteer would do, replicate those types of efforts, extract that information, and then use a volunteer to then verify did the AI actually get it correct? Did they actually, 'cause they, they'll be able to see the highlighted passages that the chat bot is drawing from. Do they actually summarize that information correctly and hopefully help us to speed the pace at which the volunteers and researchers can do their work? So I don't think that AI is replacing, and I mean I've, I've had a lot of pushback. From a lot of colleagues and fellow researchers and academics that say, no way AI can do my job better than I can. And that's, I completely agree. I don't think that AI should be replacing the jobs that humans are doing, but what I have seen it be able to do is to actually help us do our job faster. Is is there any one surprising insight or moment of clarity that have come outta these chat bots so far? Yeah, I mean I think the number one point of clarity is actually in related, in relation to the last thing I said actually really have seen that. It's really not a substitute for humans. I really think that you get a lot of, and, and so in statistics and probability, we talk, we talk a lot about like regression or convergence to the mean, and in a lot of these big kinds of data, you can see that when you get at tons and tons of data, you kind of see this convergence towards like a normal distribution or towards like a central measure, right? And I do think with a lot of these types of generic chatbots, that's exactly what we're seeing. So I don't think that that a lot of these chatbots and AI will give you necessarily the most like insightful, or most original, or the most like novel insights because it's converging towards the mean. So they're gonna pull what they can from as many texts as possible. And so that's why you're getting a lot of generic responses. And so being in academia, and I've been in academia now for quite a while, for I guess like if I count my PhD for like almost 20 years now. And so I've reviewed a lot of student work and so it's like very obvious. I can totally tell when a student has used AI because it's like things, you know, 'cause it's like I'm an expert in the topics often that they're writing in. And so it's like, oh yeah, I've heard this argument before. This sounds like very familiar. It's not anything new or like particular earth shattering that's necessarily being generated. So on one hand, I've been quite surprised at like how generic a lot of the responses have been and how that gives me confidence that we still need humans who really know the domain to really, I think, lead. So I think that's like one thing. I think secondly, on the flip side, on the corollary, I actually see how even experts themselves are not necessarily that great at distinguishing falsehoods or embellishments or, types of hallucinations that can be generated from generic chatbots. So we have a study that we did last year after we developed ChatNetZero, where we looked at two domain specific chatbots, including ChatNetZero, and then also ChatClimate, which was developed by researchers outta Switzerland. And then we looked at three generic chatbots, including Gemini, ChatGPT and then also Coral. And we asked both of the chatbots a series of 20 questions that related to climate policy or just, yeah, yeah specifically climate policy and climate action. I. And then so then we we asked them the same 20 questions. Some of them were factual, some of them were more nuanced, like, is apple's net zero target credible? Or is what Amazon doing? Is it like aligned with scientific pathways? You know, things like that. And then we took all of the responses that were generated. We blinded them and then we sent them out to IPCC and climate policy experts to then evaluate each one of the responses based on their overall quality, their factual accuracy, and their relevance. And I was really surprised when we looked at the accuracy of the responses generated from different chatbots. So the fidelity of the response generated compared to the original source document, the domain specific chatbots had much higher accuracy. So ChatNetZero could answer questions about like climate policy a lot more accurately than Gemini or ChatGPT or Coral. But then when we asked the experts, they actually preferred the responses of the generic chat bots to ChatNetZero. And that was because in order to constrain the responses and to prevent hallucination, we only allowed ChatNetZero to generate responses that were 150 words or less. And if you know, by using ChatGPT it has a tendency to be really verbose. It has a tendency to embellish on its answers. And so that happened to actually be more preferred by the experts. And so they rated it and those responses much higher than they did ChatNetZero. And that was something really surprising to me and also impressed to me the fact that we actually are not very good as humans in judging these nuances and accuracy. We are actually often fooled by these generic large language models. So that was something that also really surprised me. And then I think also the fact that we're facing a new paradigm where these AI models and generative AI are generating information at a pace that we've never experienced before. I mean, like every curve is exponential in terms of how much more data and information is being generated today, than even like two or three years ago. I mean, it is astronomical how much data is being generated, but what's not happening, and I've had lots of conversations with computer scientists and people who are developing these models. We're not doing enough on the, on the output. We're not evaluating and saying. Is what is being generated actually accurate in producing high fidelity and, and credible responses. And that's a huge problem because then you're getting a lot of information generated where frankly, as a public, we don't know what's accurate. So there's a really great book that was written last year by these Princeton computer scientists called AI Snake Oil. And they exactly point to this problem of the fact that there's a lot of snake oil that's being sold by startups and other companies that are claiming. All of this amazing application of AI and it's gonna do X, Y, and Z thing and it's gonna revolutionise, blah, blah, blah. But then when these companies then buy into, or these, these customers buy into these startups, they realise, oh, in this healthcare domain, it's actually not very accurate. It's only like 75% accurate or something like that. And so a lot of these types of models are not being tested in real world settings. They're only being developed in a very controlled environment where they're being tested and trained on a very select amount of data. And it goes back to the original part of our conversation about the need for good data. And so there's really this disconnect. And so one of the things that I'm doing in a current project called Claim, which is Climate Leadership And AI Integrity in Climate Mitigation is trying to bring together people from all different disciplines, from climate communication specialists and behavioral psychologists to really be able to understand like, how do people actually perceive this information that's being generated? How do they decide what information is credible and what, what is incredible? How can these chatbots produce output in a way that can help users really discern what's accurate and what may need their own kind of evaluation. And then the computer scientists, the AI experts and the climate policy experts in, into the same room to help really think about these challenges and what kinds of research efforts are needed to really like break down the problem and to be able to provide this type of guidance and to, I think, really lift the hood and to say you know, not all AI is the same. And we needed to be demanding these kinds of transparency on the part of these big tech companies that are developing these and frankly, blindsiding us into thinking, into, luling us into thinking that, you know, every AI response is like the same and that it's accurate. You know, I think there's a lot of problems with the opacity, with a lot of these kinds of models. So that's what we're trying to do with this new project called Claim. And if we stare into our crystal ball, what's your boldest hope for AI and climate policy for the next five years? Yeah, so I think my boldest hope for climate policy because frankly speaking, the policy world is very much behind when it comes to AI. And you know, I don't, I don't wanna offend any of my friends and colleagues who work in policy institutions, but this is just the fact, I think policy moves at a glacial pace when it comes to adopting these new technologies. And there's a reason for that. They can't be jumping on every new trend that pops up. I mean, I think it's just. It is just not possible with how decision making processes and those kinds of timelines operate. But I do feel like they need the kind of information to one, understand what it is, how it can impact their own particular processes and their own work. And then I think three, I think the most hope is how can they leverage it to actually scale the solutions at the speed that's required in order to avoid the most dangerous impacts of climate change. So my hope is that policymakers in the climate change domain can get up to speed quick enough, and we can help them through interdisciplinary collaborations really think about, okay, what are the specific use cases where it could actually make a huge difference? I don't think that it's like we need to have a what do they call like a mallet, like hammering a, a nail or something like, I don't think that necessarily we need like this huge brute force or blunt force kind of tool coming in to try to do everything. But I do think what we need is that we need to work together with policy communities to think about, well, what are the specific use cases where it could actually be really useful and really transformative and let's focus on making that happen. So that's like really my hope. And what we're trying to do in this Claim project is bringing together those policymakers, and I think there's some really easy low hanging fruit where that can be done. So we just finished in 2023, the first global stock take for the Paris Agreement. That was a three year long process in between these five year cycles where governments have to submit what they've done on climate change, what progress they've made, what challenges they face, what they hope to be doing in the future. And there were literally, I think, more than 300,000 pages of documents that were submitted in the global stock take process from companies. That's not even including all of the non-party stakeholders, so companies, NGOs, individuals, academics. We could also submit our own views on the Paris Agreement and climate policy and, and where we stand with respect to climate action. So the UN was faced with how do I evaluate all of these documents? And the thing is, is that we have these tools. We have large language models that have been around for decades, and now we have these chat tools that can help make the analysis and the synthesis of that much easier. So to me, that's a very low hanging fruit use case where we could easily take all of these documents that are being submitted. Train a model that has been fine tuned for the specifics of climate change, and then allow for policymakers to very quickly synthesise that information to understand, okay, well what are some common actions that governments have been taking? But where are some gaps? Like where are, what topics are less frequently discussed? Where actions less frequently discussed? And then where is progress really being made? How can I try to make sense and aggregate all of that? And then I think that would be a much more efficient process. And in the learning document that they submitted six months after the global stock take ended in December, 2023, they did have, they had just one sentence that mentioned AI. There was one sentence out of like a, I think it was like a 20 page document or something, just synthesising lessons learned and they said, yeah, we should look into the possibility of AI to try to help with that. So like, it was like one line, but I think a very important one line because since then we have seen these tools just really take off. And so my hope is that we can help to identify those particular pain points for them and where it's very challenging for us as humans given a short timeframe to be able to synthesise all this kinda information. And we're kind of facing that with the IPCC right now as well, where we have to then survey the state of literature, the of both peer reviewed and also gray literature to try to identify Okay, like what's been done, what is the state of knowledge on these various climate related topics, and then how can we then appropriately distill the lessons from that and then point to, and then give that to policy makers to say, okay, these are, here's clear indications from the evidence of what can be done and, and what needs to happen in order for us to tackle climate change. So to me, that's, that's my hope. Okay. And, a left field question for you, if you could have any person or character, alive or dead, real or fictional as a champion for the use of AI and data in the climate space, who would it be and why? I know you're gonna ask me this question, and to be honest, I did not prepare. This a really great question. I mean, for me, one of my climate heroes has always been Christiana Figueres. She was the Secretariat of the UN Framework Convention on Climate Change, who oversaw the lead up to the Paris Agreement, and then was really instrumental in shepherding the, the Paris Agreement to its successful conclusion in 2015. So I know she's very excited about data and she ended up being like one of the big data champions. And actually it was really cool to see for someone who's been working in the space. You know, my job that I mentioned at WRI, working on the Greenhouse Gas Protocol, that was in 2006. And I remember when I started the job, I was like, I wanna make data relevant. I want to get people excited about data. And frankly, that was hard. I mean, you would walk into a room of policymakers or business people and you would say the word data and statistics and their eyes would kind of glaze over. And I think one of the things that Christiana did was to also get people excited about data and recognising the value and just having data together in the same place. And this is still an argument that I have to make all the time in academic settings. I mean, I recently, a couple months ago a a colleague at another institution basically talked down on my work. I mean, we had dinner and he was like, yeah, your work is, I mean, he was saying this in, in so many words, but your work is not meaningful because it's just data. And you know, like, why is it that data would have this kind of dirty reputation in an academic or a policy or a business setting? It's like, no, like it's actually like I wanna get people excited.'cause we need it. I mean, we need it. And to do your fancy econometric models or to do your fancy AI, we have to have basic data. And Christiana recognised that. And so they, her team commissioned me in the lead up to the Paris agreement to actually help them make sense of the, the emergence of business and non-state actor data. And they wanted me to help make sense of it and try to collate all the data together 'cause it had never been done before. And they invited me to then actually present that information in Paris at one of the UNF ccc press conferences where they normally don't invite civilians because they were worried. They were worried. And they thought, okay, well governments cannot agree. And this is like Copenhagen Redux part two. Then we need to have something that we can fall back on. And that's like really trying to explain the extent of all of the businesses, all of the subnational governments, all of the financial institutions that are frankly in this and are supporting an ambitious climate outcome. And so for people to talk down and say, oh, your work is only data, it's not meaningful. You know, that was just frustrating and frankly not true. And so I think having like a champion like Christiana who commands a lot of respect in the climate policy domain and she has a lot of credibility and she knows the hard work that needs to be done. I think that would be really huge if she were to like give a stamp of approval and to say, Data Driven EnviroLab I think what you're doing is really great and you can keep chipping away and policy policymakers listen up because this is really important. Work with this woman. I think that would be, that would be a dream because she's kind of been in the trenches in the policy world. And so she knows yeah, the ins and outs and, and the challenges and the real limitations, but then she also knows the potential of data. So for me, I think that would be one person that I would love to have in my corner. Fantastic, fantastic. Angel, people would like to know more about yourself or any of the things we discussed in the podcast today, where would you have me direct them? The number one place, place would be our website, and that's data driven lab.org. So we have links to all of these other projects that I mentioned, the Net Zero Tracker, ChatNDC ChatNetZero. We also have a link to all of our publications, a bunch of white papers and peer reviewed publications, and a lot of our data sets as well that underpin a lot of this research. So I would encourage everyone to go there. Tremendous. Fantastic. Angel, that's been really interesting. Thanks a million for coming on the podcast today. Thanks so much for having me. This was a great conversation. Okay, we've come to the end of the show. Thanks everyone for listening. If you'd like to know more about the Climate Confident podcast, feel free to drop me an email to tomraftery at outlook. com or message me on LinkedIn or Twitter. If you like the show, please don't forget to click follow on it in your podcast application of choice to get new episodes as soon as they're published. Also, please don't forget to rate and review the podcast. It really does help new people to find the show. Thanks. Catch you all next time.