3, dash 5, 62. We're gonna do a brief description in the beginning, and then we're gonna answer questions through the chat.
So if you have questions that you're thinking about, please feel free to start typing in the chat.
Now hopefully, we'll answer those questions as we go.
But please start typing your question. So I'm gonna introduce my team here and actually, Jay, I'll start. Jay, would you like to introduce yourself?
The the program offices in the is division. The passing type is cluster.
Okay. And then India.
Hello! I'm on in the battery. I'm in size in the computing and computer foundation division.
And I'm in the software and hardware foundations cluster.
And David Gorman.
Yeah, thanks. I'm David Corman I'm part of the Cns division in size.
And I also lead our cyber fiscal system program.
I don't believe Pavitra is here today.
I think she's she was double booked, so the visa probably car is one of our program officers, and our communication and computing and communication foundation.
So an important part of our team.
Oops. So we we're oops sorry. Sorry, so we're all the important thing in this slide is really to realize that all of size is taking part in this.
So we see that there's really important aspects of each of the divisions.
So whether it's communicating and communication foundations, whether it's computer network systems or whether it's information and intelligence systems, we see aspects of all of these areas in here.
So that's why this is a cross. This whole initiative is across size, because it's important in so many different ways.
I also wanna introduce our partners or have our partners introduce themselves?
So from open fillet philanthropy projects. So Asia Cootra is one of our colleagues.
Hi! I'm Ajaya! I am a Grant maker at the open Philanthropy project, focusing on AI safety issues.
And our my partner, Dan Hendrix, is gonna be doing most of the talking.
He has a lot of experience in in the AI safety field, and he runs the center for as safety.
Yeah. So I'm collaborating with open philanthropy on this.
And I really excited to see so many people interested in safety, especially now, as the broader world is introduced in it too, and the public is wanting some safety research.
So I'm glad everybody's here, and, thanks to the Nsf.
For doing this initiative.
Yeah, thank you so much. It's very exciting.
And thanks to our partners with our partners, we don't get to do the things we wanna do.
And the things that we all think are important. So we're really very grateful to our partners on this.
So thank you. So I think, in Jan just alluded to, our AI systems are increasing.
Rapidly, and we're seeing all sorts of new things happening.
But there, we know they're deployed at high stakes, and we see this safety issue is becoming extremely important, as if it wasn't before.
But I think as we see the proliferation of AI, we really this safety issues become extreme.
And we really wanna do more you know, we we wanna make sure accuracy.
Efficiency is scalability are done. But we really wanna make sure that they are.
These systems are robust. So when we think about extreme events, we think about monitoring for stranger and safe behavior, we really wanna start designing systems that are built with this in mind that are really that are coming prepared to think about safety as one of the first features, so we thought in a safety, we should have we should
have a joke, but I from one. It's not the happiest to judge, but I think the thing we that we included it on was, but we really have to think about safety is not happening by chance safety is it's got to be a critical.
Component. So so given all this, we really wanna think about what are the undecirable system behaviors that are safety issues.
So what kind of things did they? They have? So we think about over blunders, prediction, error, system crashes, silent failures, like reporting on justified confidence levels out of distribution issues and competently achieving unintended objectives.
So we want people to start to think about these undesirable behaviors and make sure, as we deploy systems, that these undesirable behaviors are coming to pass and that they're identified before that cause a safety issue so in our learning enabled systems we have learning components that include
but remember, these are not limited to deployed system, and in health care, medicine.
We can have them in criminal justice. We can have a monomonomous and cyber physical system and finance.
Remember, as long as these in health care medicine we can have them in criminal justice, we can have them autonomous and cyber, physical systems and finance.
Remember, as long as these learning enabled systems are, are a safety issue, they can be in any one of the Dom but these also include foundational learning based systems that may be subsequently applied to many downstream domains.
So the topic we're talking about is broad. But again we remember that we're really thinking about the safety component.
So what we're trying to do really here, and working with open fill and Nsf is soliciting foundational research that that leads to learning and enable systems with which safety is really done with a high level confidence.
Now, I know that seems like it's obvious, but I think we've all seen that.
That's not the case as we're doing deployment.
So this is what we're trying to get with this.
We're trying to build the science base that gets us these high levels of confidence in these learning systems.
So we'll consider it as the success. This program really works.
If developers are the future learning and enabled systems, can one informally explain why the system can be deployed safely and then unpredictable environment, and then back these informal explanations with rigorous evidence that the system satisfies safety specifications?
So you can tell the general public why it can be deployed safely, and you can back it up with the data and the evidence for that.
So the program is looking for proposals that advance. You can do it.
This general theories, principles, and methodologies to get to these safe learning enabled systems.
And we want to get beyond the single problem. Specific problem. Instance.
So we wanted, we really want to go deeper than just one simple, one simple problem.
We want to really think about safety is this whole issue, and we've gone to think about it.
It's as we do scalability and deployability.
So ideals for the program really are that proposals have the potential to make strong advances in the design and implementation of safe learning enabled systems as well as re-advancing methods for reasoning about the safety.
So we know we need new methods. We know we need ways to think about this.
And we're hoping that this initiative really brings people with new ideas to the table.
So ideal, will demonstrate how these 2 objectives will be achieved, provide evidence that the proposed approach will improve notions of safety, and then argue that potential for lasting interest on both rigorous safety evaluation methods and on the design and implementation of safe learning
enabled systems.
So our safety guarantees here, we wanted one of the things we wanna verify that the learning system achieve safety guarantees for all possible we know that that's gonna be difficult.
But we really that's, you know. Obviously, that's our goal.
But we know that's not gonna happen. So we really wanna think about consider considerations for establishing safety guarantees, systematic generation of data from realistic but yet appropriately pessimistic operating environments, resilience to unnat unknowns so monitoring for
hazards or behaviors, new measured methods for reverse engineering, inspecting and interpreting the internal logic of these learn models, and then method for improving the performance of, or by directly adapting the system's internal logic so these are what we're looking
for. And we're thinking about establishing safety barriers.
So our safety requirements, any system claiming to satisfy a safety specification must provide rigorous evidence through analysis corroborated empirically or with a mathematical proof.
So we can do this empirically or with a mathematical proof.
But we have to provide rigorous evidence. That's important to this initiative.
Proposals also, and that increase safety primarily as a downstream effect of improving standard systems.
Performance, metrics unrelated to safety, such as accuracy on a standard task or not in scope.
We have planned of initiatives. We're improving system.
Performance is part of what we do in this. We're really looking for ones that directly impact safety.
So if you looked in our solicitation, you would see that all proposals have 4 things you have to talk about.
The notion, of end to end mathematically or empirically based safety to it.
In plain English, what do you? What do you? What are you doing?
How are you talking about this? And then justify why the end-to-end safety product properties are critical in this system.
And that's really important. We don't want downstream effects of just improving performance.
Really? What about the end? Safety properties on this identify the environmental assumption for the safety property. So in what conditions, what?
What are, where, what is the issues that we're facing?
And what are the automated, semi automated or interactive techniques for establishing the degree to which the safety project properties are satisfied?
Again, we know it's hard to satisfy all the possible options, but we really want you to come to us saying what you think you're doing there, which which degree the safety properties are satisfied, and then demonstrate these techniques techniques achieve safety and again.
it's mathematically or empirically through rigorous simulation, prototyping integration with actual learning enabled systems.
So again, when I we talk about notions of safety, we're talking about robustness and resilience the tail risk monitoring systems for anomalous.
And unsafe behavior, interpreting reverse engineering, and expecting a learn systems, internal logic or reliability under human error.
So!
So when we do the Nsf Review criteria, of course we're talking intellectual merit and broader impacts.
And those are the same criteria from the National Science Board.
So? The first question we ask of all of our panelists is, what's the potential for the proposed activity to advance knowledge and understanding what within its own field or across different fields?
We also for broader impacts, talk about the benefit to society or advanced societal outcomes to advance.
What happens? We talk about? What are the new creative or potentially transformative concepts?
Is the plan well, reason that sort of sound rationale for this.
How qualified is the individual team or organization? And are there adequate resources?
So those are a standard review curriculum.
For this one. We also say that the proposal will also be evaluated, based on the components.
So we want you to discuss learning enabled components.
The proposal should describe the components and provide research why they are appropriate for the system being stuck.
Sometimes you can't study the whole system, but we want you to think through.
What components you're studying, and why, and make the case.
Why, that's the appropriate it's it's components to test.
We also again come back to the rationale and clean language of why do the end and safety properties are critical to this in this system?
We also want the safety plan. Again, those in the environmental assumptions under which these safety properties are ensured.
So you know, we're thinking about what are the techniques there?
And what are the what are you doing? So that's our safety plan.
And then validation, so you need a plan to validate these techniques, and that might be mathematically or empirically.
It might be through rigorous simulation prototyping, and integration, with actual including sub scale learning enabled systems.
What is really important here is for the synergy projects, the validation plan must include experimentation on actual learning enabled systems.
So for the synergy only that's it must include experimentation.
So with that we can move to questions. We have a list serve.
If you need the listserv. So with that, I'm gonna open this up for questions, and we have a great team here to answer your questions.
So alright, so!
Okay. So in the.
So there's a question here, it says, is there a criterion for us to decide whether one specific application qualifies for an actual learning enabled system?
Anybody.
So I can. I can answer that for part of the give at least part of the question, especially when we think about the synergy element.
So ideally, the ideal case is, you have your research, you create an experimentation plan, and you evaluate what you've defined as safety properties on a real learning enabled system.
A. For example, autonomous vehicle, sure, short of owning an experimentation platform like a large autonomous vehicle, a sub scale, autonomous vehicle that's still includes the basic you can run your experiments.
And that qualifies, or for that, it's a system that has the sensing has the processing, and you can create experiments where you challenge your safety criteria.
Thank you, David. Anybody else want to comment on that one.
Alright. There's another question here about, does this program allow for research on existing systems, such as Chat Gpt versus developing new systems?
So do you? Wanna if somebody wanna answer that part of the question.
I mean, I think large language models are quite relevant for existing applications, and you could come up with some approved model or some modified model using using some existing large language models.
Make those safer. So we need to create a new one from Scratch.
There are many ways that we could understand large-language models better.
We could try and understand their internal logic by finding mechanisms inside of them that explain how they're doing some of their decision making to give us more confidence whether they're reasoning is appropriate.
There are other things you could do on top of Chat.
Gbt like. See if it's if it's encountering an anomalous scenario, or whether we should trust it in a different type of condition.
Those are all things you could use to study models like Chat Gp.
But you could easily place, replace it with some other large language model.
I think if we're concerned about real world applications, then the large language models are an interesting new one that are basically touching on almost every field.
So that seems to have be a particularly interesting type of model to study.
If we're trying to make a system safer in the real world.
Yeah, I would add to that that the text of the the call says that foundation models on which further downstream applications could be built are in scope, and I think large language models like Chat Gpt and others would count for that.
Yeah, I went to I, the purpose of this program is not divide by any specific system or applications, but the focus on safety issues.
So the problem can come from the existing system, or technology, or the you approach it to those other applications.
Anything you can define as a safety program that could. Where do I choose this program?
So I think that leads into the next question, how broadly is safety define safety?
Consider things like limitations to critical thinking or other more psychological harms versus harms to data, privacy, etc.
Well, so we include in it a discussion of uncertainty.
So if we can have models, have calibrated predictions, and if they're if their uncertainty is interpretable that can help people make better decisions.
So if we know when to trust their model, and when it's more likely to be mistaken, and if we can get that to be highly reliable, and that would be useful likewise, if we can have some sort some guarantees on its behavior like certifies robustness certifications that can
also, help. People know when to use it, and that it also touches on another question about how these would integrate into socio-technical systems by creating some of these safety measures and knowing knowing when to trust the model that can help a human operators more wisely to decide when to
trust the systems, and I suppose, likewise with, if we can provide explanations for how they're arriving at their decisions and are able to interpret interpret the internal processes inside of models that can also help people know when to trust these models and address social technical issues.
Hey! David!
So we can. Can I add a slightly other additional dimensions for that question?
There's line. We put language in there that says you should be able to express the safety properties in terms of playing English, and what we in part want.
There is you the panel that's reviewing as well as the program.
Directors should not be thinking of this as the riddle of the space.
It should be very clear to us from your description. What are the saving properties, and why they are safety properties.
It shouldn't be a question that we struggle with understanding.
And I suppose building on that one thing that if somebody were to say, well, we'll reduce the error rate, or something like that, and that's the safety proposal.
Well, that's a bit that's potentially too broad.
Error rates about. What is this particularly related to any harm or hazards?
Is this generically increasing the image, that accuracy or the knowledge of language models as they're answering.
You know, exam questions, that sort of stuff is it distinctly about safety?
And there are many other sort of programs for that. So I should be.
So I agree should be, and the text confirms that it should be clear that it's distinctly focusing on safety and not making models.
Doing doing this sort of research that people have been doing for the past several years, and just more of the same.
So I think this the question is, it's not only physical safety, but you have to define your safety criteria, and you have to make it clear.
So it has to be able to be understood and it can't just be improving performance unless you're dying.
That you have your time, that to safety. Is that my summarizing this? Well?
Yeah, also, I want to add there some issues.
A lot include this program like a security fairness.
I think so. Those are probably related to safety.
But it's a lot I mean, let's not include in this program.
We have made a career in the in the summer station.
On security. We do have some language about appropriately pessimistic environments, though, and simultaneously, in doing simulations or trying to come with hard examples that they'd be appropriately rigorous.
So you still still, there's certainly a hope for doing some rigorous stress tests, testing of the system. So.
Sorry I shouldn't mute my cell phone. There was a question about what's the criteria to decide whether one specific application qualifies for an actual learning enabled system.
I think the question is, is there are there we're talking about.
We're we should learn more than a single application.
But they but the learning system may be one system, right? We can have pieces of a larger system, or we can have a full learning system.
So you can imagine this in many different ways. Do you guys wanna give some example? And ninja?
I was just going to say, as the civilization says, a learning enabled system is one that has, and that is machine learning components.
So that can be, you know, it can be very broadly interpreted.
So whatever it, the system with machine learning components is going to be considered. Learning enabled system.
So chat, gpt, or ama would consequently yeah fit that bill.
Alright!
So there's a question here. It says, components on the component issue.
The proposal, describe the system components and provide reasons why they are appropriate for the system being studied.
What does the system mean? Here? The deployable AI system?
I think it could possibly be interpreted into it, so it could be that they are system, or you could imagine the larger system where there are some human operators involved.
So if we're if the model, if the learning enabled model is itself outputting things like confidence measures or an indication that there's anomalous behavior going on, or that it there's potential like, for instance, maybe a model is gaming and objective or pursuing an
unintended objective. Maybe if there are features that enable people to catch those issues and reduce exposure to those hazards that that could also improve safety.
Go ahead!
I'd add to that. Oh, oh, I'd add to that!
But one type of system, that open plan to be has particular interest in is that these large language models are starting to assist in coding workflow.
So they're starting to write code at the request of humans which humans then run or do other things with.
So I think it would, thinking about how the outputs are used.
Are they just gonna be run? Are they gonna be monitored in some way?
Could be construed as like the system that this machine learning model is part of.
Yeah, as humans are being more and more removed from the loop.
And as people are using AI systems to replicate a lot of that functionality, the notion of what the AI system is, seems to be expanding, and it seems quite important to keep track of how it could be creating new new sorts of problems for Us.
As as it replaces a much functionality.
The next question is the notion of fairness, not being able to provide recourse considered as safety.
So there's the notion of safety more broadly, but then there's what was in the in this solicitation.
So of course, there's the problem of alignment to human values, things like well-being and impartiality, people getting what they deserve and what not.
But in this solicitation there's generally more restricted to deal with to deal with more standard harms and hazards.
There are generally other proposals and programs that get an issue such as fairness, though.
So this isn't to say that this is some like really strong intellectual stance.
Against against other sorts of human values primarily, that there's only so much the program can fund.
So I. The next question is, when learning enabled systems are embedded in socio-technical systems, so just some of the examples in the safety is arguably not just about the behavior and the technical system, but how it's used by people how much of this is in scope, or is this intended to be
purely technical.
Well, it's partly mentioned before. There are plenty of technical things that can be to help with the socio-technical aspect of people not unduly trusting these systems.
If we have a better idea of their internal workings, or when they're likely to be useful or when they're likely to succeed or helping human operators monitor these systems more effectively, all those help with very various socio-technical aspects so there are plenty of there are
plenty of technical safety measures one can do to improve broader systemic factors that affect whether there'll be accidents or it could be caused by these systems.
I I think that we're really looking in this one.
It's more of the technical than it is of the societal.
So I think that.
Yeah, I as a that, basically, we have, you have to be clear.
Define the problem mathematically, or experiment to me right?
So you needed to create, defined. So technically, sometimes they are easier to define.
Of course, if you can come up with a definition for the first, for other, we are open to any safety problems.
As far as you can define community, and you have the way to verify it.
That's what you find.
So, yeah, the solicitation. Okay?
Yeah. Go ahead.
Oh, thank you. The solicitation says that you need to define the particular safety property you're going for in plain English, and that property can make references to how the system is used, so you can say, even if a user uses the system in in X reckless way the system should not do why
bad thing that that can kind of make reference to the socio-technical aspect.
But then the particular solutions, like Dan said, are are, and Wendy said, are more of the like technical solutions to address.
Yeah.
And and as well in the word, in the wording. There's end to end system, too.
So we're, of course, trying to think about how it touches on the touches on the system. In context.
Oh, if I can add, one has to be very clear about what what are safe behavior?
And the system, and then 1 one way of thinking about that is is just, you know.
If if you think about the semantics of the system, what are the traces?
What do the traces of that system look like? What are the safe traces?
What are the unsafe trade? What is? Yeah? So these are issues that on may consider.
Okay, we've got a lot of questions about the specific definition of safety.
And the question is, is it just physical harm?
And I think we've gone through that. But.
Do we want to add anymore the definition of safety?
So safety isn't just making the system do perform better.
There are many ways in which you could have systems, have fewer errors, and that would in some ways make this system safer, but then they also might be more capable at doing things that are potentially hazardous.
So, that that's why there's a specific, a text in the in the solicitation stating that if you're primarily just improving safety as a downstream, effect of improving the general performance of a system it's generally a rate, and things like that that sort, of thing is not in
scope so safety isn't just making the system work better.
You need to show how you're specifically focusing on some targeting safety and not targeting.
It's general overall, diffuse performance.
Anybody want to add to that?
Just simply, you know, agreeing with Dan. Yeah, really, think about the plain English element we're asking you, you know, the proposer to tell us what is safety in your in your mind.
What are you, considering? Safety and provide a wow. Reason easy to understand.
Discussion of why that is safety, and then be able to show, either through imperical or mathematical, that your system enables, proving that.
Yes, I'm meeting these criteria. It is a safe system.
Hey!
And you know, in part that's the end-to-end part that Dan touched on is really, you may.
Only what we don't want is you to consider only a single component of an integrated system.
You may be other reason here is that behavior of this component.
But how does that component, then, influence the overall system?
Safety!
Thanks. Thank you all for that. One of the questions here is so an actual learning enabled system doesn't have to already exist in the world, but can emerge from the project.
I think we're not looking for a new system. We're looking for a safety.
These are really targeting safety. So if you're gonna create a new system, and then you're gonna have to build it.
So it's not safe. It's gonna take a lot of twisting and turning in your proposal to get there.
So you might wanna pick a learning enabled system.
And then you can address the safety and and spend the proposal and the funds working on the safety issues.
Not building a new system. So I'm sure there's a possibility.
But I think it seems like a lot of extra work for not getting to safety.
So so is AI alignment considered in scope for the solicitation.
Well, there's there's some text like studying whether models are competent to achieving unintended objectives.
So what we could imagine, say, using a large language model. That's pursuing some that's taking some actions, and maybe some text based environment and maybe wanting to study some of its safety properties of how it's interacting within its environment and is it causing harms in that environment that might
be an interesting that might be an interesting microcosm that would touch on, that would touch on alignment issues in some capacity.
Okay, so we do have a question about, is it?
So there's a question about, are we we looking at safety at the level of systems, or are are we looking at individual system, misbehavior, or at a broader scale than that?
It's hard to translate this question. So.
The kind of the wholesale harms of these learning systems to society, and I will start with, as we're asking you to describe your notion of safety you're gonna try to do all of society it's gonna be hard to be able to verify that to test
that what we're looking for is, generalize little information that we get do in these learning enabled systems.
So we want you to come up with new methods, new ways to look at.
This new ways to understand it. Then we could be able to think about across systems.
You all. Wanna add to that?
Alright. Could you all elaborate on the high stakes, scenarios?
The program has in mind.
Well, there. There are many high stakes scenarios right now.
People are using. For instance, these chat bots for things like medical advice, or but what can imagine more competent systems later performing sequential decision-making.
And you would want to make those safer too. So those might be who exams, if like, for instance, with like Chat gpt plugins, and whatnot that is able to actually interface with the world, and that's creating some some potential hazard.
So since people are actually using these systems, machine learning systems in the real world there and coming to depend on them increasingly and outsourcing more diseases, making to them.
I think there are many ways in which a large language models are becoming more safety critical.
And I know we have sent Chat Gpg. Many times, but that is not a limit to this program.
Okay.
So we're not gonna say it anymore. But you understand what we're talking about. So.
But we remember we're talking about these broad learning enable systems, so large language models are one.
But there are many different things you can think about in here.
So keep in mind. We have a broad thing, and actually gets to the next question, does any does a learning and system have to have a physical component?
Or can it be entirely algorithmic?
I think there are many relevant learning enabled systems that are not having actuators out indie the real world that are still nonetheless, impacting, impacting it through humans.
So it doesn't seem strictly necessary.
Right, and to add to that disability specifically talks about identifying the learning enables system, and that needs to be there.
But you can have other things around it, like the physical system, or human in the loop.
So these are all optional things. But what we really need to see is the learning enabled component.
Hey!
Thanks very much, and welcome to. We introduced you. Thanks for answering me.
So does it do. The proposed projects have to demonstrate applicability at the proposed frameworks on more than one exemplar.
That might depend on how broad the exemplar is.
But generally there's a prioritization toward having a very general insights that are applicable to lots of different systems as opposed to a narrow, very narrow, specific application.
Remember in the synergy we're gonna ask for an evaluation.
So you really do want to think about the broader ideas, how they can be evaluated.
Yeah, this will also that depend on the definition. Right?
So what's your claim? Safety, your problem, then? Basically, you have to test it in your domain.
So, if you only claim one domains, and you can only need to tie them on one domain.
If you claim this, both generalized approach you'll have to test. But this is the 2 or more verify your system.
Thank you. Jay.
For all of you to keep asking me if we're gonna have the slides available.
The answer is, yes, they will be on the website, and the video will be recorded, and we'll be on the website too.
Is there notion it? Okay, here's the safety question.
Is there notion of safety subject to interpretation with respect to the system under consideration, or is there an objective notion?
We must adhere to.
It's the answer is you're providing us that notion of safety.
However, it has to be a notion that shouldn't be such a.
Notion that no one on the either the panel or program directors really understand should be clear to us.
Thank you so can you elaborate? There's a question on this.
And Jay mentioned this earlier, but it's on the security aspects.
Finally research on securing learning and enabled systems against adversaries is not in scope.
David, you want to explain what we were, that one.
I know you mentioned earlier.
They also meet.
Yes.
Yeah, it's, for example.
So people ask for the for the light worker, security.
So those problem is not included. Right?
Also asic problem fairness problem. So that's we have other Nsf program to just address those specific problems.
So for this question, we focus on the safety. So also, safety here is quite broad.
So you have defined, and to tell us what's the safety you train English, and then you you can verify it.
But we not talk about the fairness. We don't talk about network security, and we don't talk about Asc.
Yeah, yeah.
We do talk about deployment in very challenging environments and situations, or in the face of size, of some very pessimistic hazards, though.
But yeah, things like network security or intrusion. Detection is is generally covered in other programs.
So!
Yeah, he has. The security is a specific means, is a networking.
Those kind of a security attack.
Could it proposal be theoretical study with simulated experiments?
I think. Yes, the solicitation asks for that could be a theoretical study with experimental validation of the results. Yes.
Only question there is when you say simulated to me, that becomes more.
The less the property, less of this synergy element, then the and that first part, first element of the solar station.
It's hard to argue that experiments on a simulated system are and a meet us.
So we can do the smaller awards on a completely on a theoretical with simulated experiments.
Send her, just stick.
But when we get to the synergy we're really looking for a validation plan for it.
Right.
So so somebody was asking, you know, if we're gonna do that, should these systems that are deployed, they said in industry. But it could be just deployed anywhere.
So if we're talking about simulated systems and simulated data, that's enough for the smaller awards.
And then what we're talking about is having in the synergy level that these are deployed systems that you're doing.
Or a subscale representation.
Are we considering human safety or system, safety or both?
Well, I think it. The solicitation speaks about safe systems.
But the way systems interact with humans could introduce safety concerns that could potentially be addressed.
So, for example, a thing that would be in scope is, if you wanted to talk about, do a project on system honesty like, how can an non-expert human tell if an AI system is telling them false information when the AI system might have more familiarity with that a certain domain like
coding or medicine than that human might be something that has a human component to it.
But then the research project would involve like, how can you have guarantees that that system is not going to live? The human.
And if you're gonna do something like that, make sure you have ground truths for what line to humans do because it medicine.
There are so many different outcomes that are possible, and finding out what ground truth is.
Often a challenge, so make sure when you're thinking about it.
You're thinking that way. Make AI library safe, not crash!
Is that in scope.
I'm not sure but my guess would be no, because the libraries are like like code written.
Maybe 2 train ais, but libraries aren't themselves.
Learning enabled systems like but the library isn't itself doing learning, or it's like more.
Just some software somebody has written that might then go interact with learning enabled systems. But.
Yeah, I think, making a good point, you'd have to argue the safety question.
And so you might have a library that you feel like makes this.
But it doesn't seem like it's from the outset when when we're just talking about it.
Now it seems like it's out of scope. But if you felt like it really was, you can also send us a one page project summary that we can give you feedback on too.
And so we are encouraging everyone to do that. To send a one page summary.
Or sub scale, representation.
When let me also add, though, and you know, if you have trouble under, if you, if you're not convinced that it's safety, think of that!
It from the panel perspective. Yeah, they're gonna be one of the ultimate judges.
And if it's not, if you have your doubts, yeah, don't depend on the panel to not have system downs.
I think that's always a good point. If we can't answer it easily, it's gonna be hard for the panels.
Exactly. That's exactly my point.
So one of the questions here is is the solicitation targeting only deep learning components are also on classical learning.
Components, like networks.
Yeah, I need any components. So we don't really specifically for any based authority or system. Right?
So the people can use any technology. You you can even come up with.
It's a new approach to or design principle that I can.
I can design the safe safer than mean systems.
That's the way we find, too. So I started thinking, you can verify it.
That would be good.
Though there may be some headwinds if you're trying to study solely support vector machines or something like that.
And because some of them may have less applicability in many applications, so there might be some tension there in not doing a deep learning systems, there might be less applicability.
And that might be hard to argue, for its broader impacts.
There is a question on the process which is when I can answer.
If somebody says I've never applied with with with a partner agency when it's Nsf.
And open fill in good ventures, so is it different, and is it different?
Not a whole lot different, but our colleagues that are here now will also be sitting and serving Pamels, and then, when, as we're making decisions, they will also have.
Input but remember, this is a joint project. So, and it's live by Nsf, so what it looks like from the inside.
It'll look like Nsf.
But in partnership with our colleagues, so.
So there's a question. Is it limited? What the means of learning and enable systems are interested in the call?
Is it limited to long, large language models like, or does it come on deep learning based systems?
I think we just.
Yeah. We kinda answered that like multimodal systems seem interesting, too.
And there could be not in deep learning ones that a lot of the a lot of applications are deep learning, and so.
So, what's the composition? In the review panel for this?
It will depend on the projects there will be. I don't know.
And then did David Jay for me, or whoever wants to win.
Oh, I think it will. It will be. It will compose us people who are researchers who are experts in learning machine learning as well as the social formal techniques and other experimental techniques.
And implementations of these systems. So it's it will be a mixed file, a paddle with mixedx.
Okay. And so what's the difference between the requirements between the foundation projects and synergy projects?
Can 2 researchers from different institutions of life in the same project has Co. Opis.
Are you simply talking about collaborative proposals?
I think there's a question about foundations versus Synergies.
But then there's also a question about I think it's collaborative.
So yes, you can do collaborative projects.
Yes, you can do collaborative. And so the other is.
The other question, and one pi applied for both.
Yes.
I think the answer is, yes.
Bye!
But that's a that's an imitation that basically each time you can only apply for one category for 1 one proposal in total.
You will only apply for 2. Right? So that's what the.
So if you feel pi this time for both categories that you're done so, you cannot apply next time.
So if you apply for one category this time, you can probably use the category next time, so don't.
Okay.
Does safety include consideration of age and contacts, such as what systems are used in an educational center?
So really it there depends. What? What are you defining as the safety properties that you're going to evaluate within the context of that setting?
There's also a question of whether fairness which I believe we've answered before or privacy are they?
Are they safety concepts?
So, since there are limited things that they could find things like fairness were generally more out of scope.
But when instring safety more broadly, you know there are obviously many parts.
It's a broad section. Technical problem involves governance involves security.
There's a lot of things to make makes us a safe.
But in the this program some of are getting many of the areas that were neglected in research previously are not covered by other programs.
So. Hence they're generally not as or it's not really in scope.
Yeah, so privacy research, many covered by this trustworth computing program already in side.
So I think that's pretty much out of scope.
This program.
It would just be difficult to argue privacy as the primary safety outcome of the project.
That would. It'd be a tough argument.
Alright! Dan mentioned, uncertainty, calibration. Can we use?
That as the only safety criteria in the program, or is it too general?
That's obviously going to be left up to the panel.
Maybe one would want some more specific stuff or things to make it have more of a safety flavor you could potentially argue that we're wanting to have certainty about when the models are applicable, or when they or maybe when there's some hazards in
the environment, or if or if the machine learning systems themselves are creating hazards, and we detect that.
So might want to add more resolution to exactly what you're going after.
Instead of using just a a broader buzz word like uncertainty.
And going off the language in the solicitation to describe the safety properties. In plain English, maybe you would want to draw out what onsafe things might happen as a result of the model being miscalibrated in a particular way.
Yeah, thank you. There's quite a few questions coming through on.
Is this related to Hci, why isn't this an Hcc.
And our human centered computing. And I think one of the questions.
One of the questions that that comes up here is so.
This is not Hcr and human centered computing group does lots of works about issues of adverse events and things happening.
But they're really documenting them, and they're really documenting what's happening.
This is really to prevent. So we're really trying to create systems here, create tools, create methods and techniques that allow these systems to be safe it isn't documenting that they're not safe.
It is really trying to address the safety issues upfront.
So, if you need to do Hci type methods to address safety, then that's in scope.
But make sure that you're addressing the safety.
How many, I don't think how many pages are appropriate for describing the components.
Rationale, safety, plan and validation are these separate sections from the main technical component are included in the technical part.
Yeah, 15 pages in the project. Just description. To give us a complete picture of what are the safety properties?
How are you going to but experiments? Are you going to prove it's up to you as a proposer to decide how you allocate?
That's page space. As you think about this. Also, think about what make what will make it easier for the reviewer to find the information that they're trying to identify from the solicitation don't hide information.
Make it easy to review.
Alright, so I hate to say it, everyone but we. We were so busy answering questions.
We ran out of time. So I did. I just looked down and realized it's 2 Pm.
And we were still excitedly answering questions. If you are interested in sending in a one page summary, please follow the emails in the solicitation.
Send a one page. Project summary talking about what you're learning angled system is what your notions are and what you plan to do.
And so you can get feedback on your projects. I wanna thank open fill and good ventures for being a colleague.
Here they are, supporting, they are coasting with us and again, as we noted, they'll be observers in the review. Multiple questions about this.