I don't know that I'd say _useless_ but there's _a long way_ to go. I also suspect that many developers are going to wind up being dependent upon them at the expense of getting familiar with reading code/documentation, exploring with debuggers, writing forum/blog posts, etc.
Here is one example from the last time I asked Claude a question about message filtering on AWS SQS, which is a very common exchange in my (relatively limited) experience with these tools.tool
# me responding to a suggestion to use a feature which I was pretty sure did not exist
> ... and you're sure this strategy works with SQS FIFO queues?
# Claude apologizing for hallucinating
> I apologize - I need to correct my previous responses. I made a mistake - message filtering with message attributes is NOT supported with FIFO queues. This is an important limitation of FIFO queues.
If I didn't already have familiarity with this service and its feature set, I would have wasted 15/30/60 mins trying to validate the suggestion.
I can't imagine trying to debug code that was automatically generated and inserted into my toolchain using Copilot or whatever else. Again, I'm sure this will all get better but I'm not convinced that any of these tools should be the first ones we reach for.
Here is one example from the last time I asked Claude a question about message filtering on AWS SQS, which is a very common exchange in my (relatively limited) experience with these tools.tool
# me responding to a suggestion to use a feature which I was pretty sure did not exist
> ... and you're sure this strategy works with SQS FIFO queues?
# Claude apologizing for hallucinating
> I apologize - I need to correct my previous responses. I made a mistake - message filtering with message attributes is NOT supported with FIFO queues. This is an important limitation of FIFO queues.
If I didn't already have familiarity with this service and its feature set, I would have wasted 15/30/60 mins trying to validate the suggestion.
I can't imagine trying to debug code that was automatically generated and inserted into my toolchain using Copilot or whatever else. Again, I'm sure this will all get better but I'm not convinced that any of these tools should be the first ones we reach for.