More observations about LLM coding help

(This will be a nerdy post about .NET development.)

I've spent some time over the last few days experimenting with GitHub Copilot's abilities to handle some specific programming tasks. Specifically, I asked a few of the models to do these tasks:

• Examine all of the projects in this solution and add the parameter "CancellationToken cancellationToken = default" to all async methods, with appropriate XML code comments.
• Now propagate the cancellationToken values to all method calls in the updated methods.
• Now add "ConfigureAwait(false)" to all async method calls in the following projects...
• Review the solution and recommend performance, security, stability, and usability fixes.

Here's what I found:

Chat GPT 5 can do all of those things well. But I only get a handful of premium credits per month, so after the initial test, I just used it for its recommendations, which were thorough and well-explained.

I only tried Claude Sonnet 4/4.5 for the last prompt, and it had most of the same recommendations. It's worth having both models look at code as an outside reviewer. I didn't try Claude on the refactoring tasks, because CGPT5 did such a good job.

Chat GPT 5-mini struggled quite a bit. It was dog slow, taking 15 minutes to execute the first prompt on a 7-project solution (not including test projects), and it could not figure out how to resolve some ambiguities. For example, seeing a method with this signature:

public async Task SaveAsync(T item) where T : class {}

instead of adding the cancellation token parameter to the existing method, it created an overload:

public async Task SaveAsync(T item, CancellationToken cancellationToken = default) where T : class {}

and then changed the original method to

public async Task SaveAsync(T item) => await SaveAsync(item, CancellationToken.None);

which required manually removing all of the single-parameter methods entirely. It also munged up a lot of the XML code docs, removing entire blocks of it for no apparent reason.

Falling back on Chat GPT 4.1 was the solution: it doesn't "know" as much as its younger sibling, but it works nearly as fast when given clear parameters.

I'll have more later, but right now, I need to get Cassie to the vet for her annual checkup.