Building Declarative Agents with API Plugins for Microsoft Copilot Part 5 – Tips and Gotchas

As we wrap up this 5 part series, I wanted to point out a few things I came across building the solution you’ve been reading about. Hopefully they will help you building your own Declarative Agent (DA) or get you unstuck.

In part 1 I talk about what Declarative Agents with API plugins are and how they fit in to the Microsoft 365 developer stack alongside other technologies.
In part 2 I cover the prerequisites required and the work you need to undertake before you can start.
In part 3 I introduce my demo application, and walk you through it.
In part 4 I show you how the demo application was built, walking you through the process so that you can build your own Declarative Agent with an API plugin.

Developer Mode

When you are chatting with your DA, turning on DA will give you insight into which API methods Copilots evaluated, which (if any) were chosen, and the HTTP status codes of calling them.

To turn on Declarative Agent Developer Mode, type -developer on (including the dash) as a single chat message. You’ll see a response indicating that developer mode is turned on. After that, messages from your DA will include a “Show plugin developer info” section, which can be expanded to reveal useful information:

Treat Copilot input like user input

This was an interesting one.

In traditional development an API would typically be called by a trusted front-end. The front-end would usually be responsible for doing the bulk of the validation work, enforcing the long-held developer maxim to never trust user input. For this reason, although APIs might have some input validation there’s often a working assumption that the data it’s getting is “clean”.

With Declarative Agents, Copilot is deciding to call your API, and is also deciding what values to use for each parameter. This might be based on what the user said, or other system prompt rules you have in place.

Your API should validate this data as if it came directly from a user!

As an example, I had an API POST call which required an item’s ID and category name, and which then updating an Azure Storage Table based on this information. There was another API method – a GET – which returned all the items in the Storage table, complete with their IDs and category names.

Between Copilot calling the GET and showing the user the list, and then calling a POST method to update one of them, it decided it would change the category name of one of the items. Nothing big, just a slight re-wording, but enough to mean that the correct record wasn’t being updated because now it didn’t match. No errors, no HTTP status messages, just a lot of head-scratching and debugging to figure it out. Nothing on the API side was validating that the category name being provided was a valid category name, or that the item being updated actually existed. Unless there had been a programming error, this is the sort of logical error that would never have happened with normal procedure programming, but is a new type of error inherent with using non-deterministic reasoning as part of a user workflow. One to watch for.

You might need to change the shape of your APIs

Depending on when you read this and also of the complexity of your APIs, it may or may not be a problem. You might hit some roadblocks in the journey of creating an OpenAPI spec file and then importing that as part of creating a DA. The errors will likely be unhelpful and not specific, but will because some part of the spec file you’re using isn’t supported by the parser used to ingest the file for the DA. I found problems with particularly complex POST bodies, for instance.

This where having an abstraction API, as we discussed in part 2, can really help, because you can change the shape of the API to keep everyone happy.

As I say, things are improving all the time (none of this even existed a few years ago!) so I expect this to soon become a non-issue, but there are other good reasons to keep an abstraction API around.

Responsible AI (RAI) will get you, probably.

Responsible AI (RAI) is the process Microsoft have for ensuring that users of Copilot have an experience that is safe, trustworthy and ethical. As part of that each time your app is provisioned it passes through a RAI gate, and if it fails then your app will not be provisioned.

For the record, I think that this is a very good thing that Microsoft are doing, and will help ensure that users only ever have a positive experience interacting with Copilot.

However… the RAI trigger does sometimes seem a little sensitive. For ClothesPilot, I was trying to generate images showing someone that looked like me, wearing the clothes that were selected, in an environment similar to the one I was in. I thought that this would make for the best comparison pictures.

I started out with a system prompt something like this:

When generating images, show Tom in a modern home-office setting. He should be depicted as a middle-age man with white skin, clean-shaven, trim build.

I started getting RAI errors, and it took me a while to work out why – whilst the provisioning process will error and tell you that you have failed RAI, it won’t explain why or what to do about it.

By the time I had finished removing things and testing in a trial-and-error way, this was the prompt:

When generating images, show him in a modern home-office setting.

The generated images aren’t as good, but any more detail becomes too stereotypical, apparently.

Conclusion

Coming to the end of this series, I’m pretty excited about the power of Declarative Agents. I think as organisations come to be aware of them, they will find areas of their business processes that can really benefit from what they bring.

I hope this has been useful. I’m looking forward to seeing what Declarative Agents you make – get in touch and let me know!