Building Declarative Agents with API Plugins for Microsoft Copilot Part 3 – The ClothesPilot Experiment

This is the third in a 5 part series:

In part 1 I talk about what Declarative Agents with API plugins are and how they fit in to the Microsoft 365 developer stack alongside other technologies.
In part 2 I cover the prerequisites required and the work you need to undertake before you can start.
In part 4 I show you how the demo application was built, walking you through the process so that you can build your own Declarative Agent with an API plugin.
In part 5 I cover off the gotchas, tips & other things that I learnt along the way.

If you’ve been following me on LinkedIn since the start of the year then you might already be aware of ClothesPilot.

ClothesPilot started as both a useful training tool and an experiment. I needed a problem that could be well solved with Declarative Agents and APIs that was realistic but without using actual customer problems. It also had to be relatable: people should be able to draw a line from what the demo is doing to their own organisation, its data and problems.

The Problem

Like, I suspect, many of my readers I’m not overly fashion conscious or aware. I don’t give much thought to what clothes I put on in the morning. A common trend I’ve noticed is that I will regularly wear the same 7-10 day’s worth of clothes, because these are the ones that have been recently washed and that I’ve put away, meaning that are “on top of the pile” when I come to make choices.

My wardrobe contains other clothes that I have amassed over the years, including some I really like. However, if I’m not careful these clothes can go for years without being worn, either because they’ve been forgotten, or because I’m not quite sure what to pair them with.

The Solution

Maybe this is a problem for AI? In my ideal world, my clothing choices would include:

• consideration of the weather
• consideration of the activities and meetings I have scheduled that day
• consideration of what clothes I actually have (it’s no good being advised to wear something you don’t own!), including which of these have recently been worn and therefore are probably being washed
• consideration of “style” (whatever that means!)

The first 3 of these things could be solved with APIs. I hoped that the fourth could be achieved via a LLM.

And so, the idea for ClothesPilot was born!

This is great use for a Copilot Declarative Agent with an API plugin. I can communicate via the Copilot Chat interface rather than having to go elsewhere, so I’m more likely to use it regularly. And with the API plugin for Declarative Agents not only can I get information about the weather, my schedule etc, but I can also issue updates to store what I’m wearing, which is crucial for not recommending the same thing 2 days in a row.

The Setup

I already had an API setup for the weather (I use the excellent API at openweathermap.org).

Calendaring was a bit of an issue, because (like many people) I have more than one calendar. For this project I decided to use my primary work calendar (Microsoft 365) and our shared family calendar (Apple). To keep things more simple for this project I decided to write my own API abstraction which would look at both calendars and return a single unified view.

Once I’d decided to create an API to abstract the complexities of the calendar, it became obvious to extend it to cover all the workloads. By abstracting the weather API call I could make it so much easier for Copilot, with a parameter-less call to GetTomorrowsWeather, with my abstracting API working out the date, setting the location etc.

I also needed an API for the last part of the API puzzle: the wardrobe. For this, I spent a dull weekend adding all my clothes into a basic Azure Storage table, together with a datetime field for when they were last worn. I then wrote an API method to return all items which hadn’t been worn in the last 2 weeks, and a method to update the table marking specific clothes as worn by setting the “last worn” date.

I didn’t have to combine all my APIs into one single API for my Declarative Agent (DA) (it can work with multiple APIs) but I chose to, and you may want to consider it. For me, it meant that I could remove complexity from the DA setup and test everything separately first. As I was working on my DA setup it also meant I gained a lot of visibility into what was happening, via logs and tracing.

I wrote my API in Azure Functions, with a API Gateway instance fronting it, primarily for the additional logging, but also to solve a limitation that DAs had around authentication methods (which no longer exists).

What’s it like to use?

We’re going to cover the setup and configuration of the Declarative Agent itself in the next blog post, but I wanted to show you the outcome of the whole process, so it feels more real.

As I write this, I’m a little over halfway through a 30 day experiment using ClothesPilot. For a month, I have committed to asking ClothesPilot what I should wear the following day, and then wearing those clothes without changing anything. This was certainly a risk as I wasn’t at all sure how good it would be at pairing clothes, colours, and styles.

Each evening, I sat down and asked ClothesPilot the same thing: What should I wear tomorrow? and Can you show me what that would look like?.

Here’s the very first day of the experiment:

For completeness, each day I took note of the AI generated image, and also how I looked wearing the suggested clothes:

There are many other examples like this, and I’ve made some changes along the way. I’ve put some more examples at the end of this blog post, but you can see them all on LinkedIn.

In short, it’s working great! It’s doing what I hoped it would do, which is to make me wear more of my clothes. Its choices have (mostly) been good ones, and I really like how it explains its choices with reference to my calendar and the weather. I’ve already decided that I’m going to carry on using ClothesPilot after the experiment finishes, and I might even try and build it out a bit!

Just as was editing this post I had an idea: I should ask Copilot to describe itself for this blog post based on its own configuration. Here’s what it output, which I think is a pretty good summary:

ClothesPilot is an innovative Declarative Agent designed to assist users in selecting their daily outfits. By leveraging knowledge of the user’s wardrobe, calendar, and the day’s weather, ClothesPilot provides personalized clothing suggestions. This agent was created as an experiment to demonstrate the capabilities of building and registering agents to extend Microsoft 365 Copilot’s functionalities.

Features and Capabilities:

• Wardrobe Integration: ClothesPilot has access to the user’s wardrobe, allowing it to suggest outfits based on available clothing items.
• Calendar Awareness: It considers the user’s schedule for the day, ensuring that the suggested outfits are appropriate for planned activities.
• Weather Adaptation: ClothesPilot takes into account the day’s weather forecast to recommend suitable clothing, ensuring comfort and practicality.
• Personalized Suggestions: By combining wardrobe, calendar, and weather data, ClothesPilot offers tailored outfit recommendations that users can follow without question.

Why It’s Great: ClothesPilot showcases the potential of Declarative Agents in enhancing daily life by automating routine decisions. It simplifies the process of choosing what to wear, saving time and effort while ensuring that users are dressed appropriately for their day. This innovative approach to personal assistance highlights the versatility and power of integrating various APIs to create practical and user-friendly solutions.

In the next blog post in the series I’m going to showing you exactly how I built the Declarative Agent itself, what prompts I used and how I setup access to the API.

Some more examples:

---

---

---